You are not logged in.

#26 2020-10-21 07:21:03

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

^ thanks.
It looks good.
Tor exited after 7s. What happened in that timespan?
Did you see a window with formatted text but 0 results?
Did you see a python error message?
Something else?

Martin wrote:

Too much work but testing again with newly downloaded code.

You know you can just

git pull

right?
Maybe you need to clone the whole repo first.

In case you're getting 0 results or a timeout, try this:

youtube-search-tor -c ''

(by default, the script restricts the choice of tor nodes to Europe, You can switch that off with -c '')

@damo: thanks. Good to hear.

Last edited by ohnonot (2021-07-04 09:13:04)


Give to COVAX! Here or here. (explanation)

Offline

#27 2020-10-21 10:42:13

damo
....moderator....
Registered: 2015-08-20
Posts: 6,663

Re: Search youtube without a browser

I am getting random(?) failures with search results, eg:

:~ $ yt-search.py bunsenlabs
Search string: bunsenlabs
1 ----------------------------------------------------------------
  Title:       BunsenLabs Lithium | Installation and First Impress
  Description: BunsenLabs
  Duration:    26:44
  Url:         https://www.youtube.com/watch?v=hdAHQvwsm-o

***etc***

9 ----------------------------------------------------------------
  Title:       System Setup & Product Review - BunsenLab's Helium 
  Description: In this video, I give 
  Duration:    21:06
  Url:         https://www.youtube.com/watch?v=O-FzwBOq2ds
------------------------------------------------------------------
Results can be found in /tmp/yt-search.py.log
------------------------------------------------------------------
Your choice (1-9, q to quit, s to search again): q
:~ $ yt-search.py bunsenlabs
Search string: bunsenlabs
No results.

Sometimes I can get results in one terminal but not in another open at the same time. Sometimes using "q" and repeating the command works, but not with "s".

I am using the most recent youtube-search from a git pull, and I'm not sure how to troubleshoot this. A timer issue somewhere?

My wrapper script for use with gmrun works the first time (usually), but no results are shown if it is  repeated soon after. Waiting a few minutes sometimes works hmm


Be Excellent to Each Other...
The Bunsenlabs Lithium Desktop » Here
FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#28 2020-10-21 14:08:57

Martin
Member
From: Stockholm, Sweden
Registered: 2015-10-01
Posts: 546
Website

Re: Search youtube without a browser

ohnonot wrote:

^ thanks.
It looks good.
Tor exited after 7s. What happened in that timespan?
Did you see a window with formatted text but 0 results?
Did you see a python error message?
Something else?

This is all I get in the terminal:

------------------------------------------------------------------------------------
Enter search terms for youtube (minimum length 3 chars): 
GC32
Timeout reached, no response.
No results.
------------------------------------------------------------------------------------
Results can be found in /tmp/youtube-search.log
------------------------------------------------------------------------------------
Your choice (1-9, q to quit, s to search again):

Nothing else.

ohnonot wrote:

You know you can just

git pull

right?
Maybe you need to clone the whole repo first:

git clone https://notabug.org/ohnonot/stuff

In case you're getting 0 results or a timeout, try this:

youtube-search-tor -c ''

(by default, the script restricts the choice of tor nodes to Europe, You can switch that off with -c '')

Sure, I know about git usage (a little).

youtube-search-tor -c ''

makes no difference.

/Martin


"Problems worthy of attack
prove their worth by hitting back."
Piet Hein

Offline

#29 2020-10-22 18:49:49

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

I made some major changes.

I spent way too much time debugging these 2 scripts (and a few more scripts for testing)...

Here's the thing(s):

youtube-search-tor is now mytorsocks.sh (shell wrapper)

This script did not work with the -c '' option, sorry about that. Additionally it was stupid that the command executed is hardcoded.
This script is now completely superseded by mytorsocks.sh, which opens a tor connection with a few configurable parameters (port, entry/exit nodes), then opens any command you want with torsocks pointed at that port.
Torsocks is "a wrapper between the torsocks library and the application in order to make every Internet communication go through the Tor network." This obviates the need for the command to be able to connect to a socks proxy, and I have removed that bit from the youtube-search python script. Hooray. Torsocks also adds some extra security.

This means you can use this script for any command. But it also means that you have to perform youtube searches like this:

mytorsocks.sh [options_for_mytorsocks] youtube-search [options_for_youtube-search] 
# or
mytorsocks.sh [options_for_mytorsocks] /path/to/youtube-search [options_for_youtube-search] 
# or
mytorsocks.sh [options_for_mytorsocks] ./youtube-search [options_for_youtube-search] 
youtube-search (python script)

The python script does not use youtube-dl at all! It (rather crudely) parses the page one gets when requesting a search from YT. There was something wrong with it - I can only guess that YT periodically changes its page layout, and the parsing broke. I have now expanded it to two consider the "old" method as well as my "new" method.
It seems a lot more robust now, so please try again.

____________________________________________________

Sorry for the hick-hack, hopefully I have settled on a final solution now.

Please review the helpful '-h' output for each script.

I have been testing so much that I had to use tor for all requests, my IP had gotten locked out of youtube.
Requests via tor now work perfectly for me!
But I had to make some changes to the python script that affect all users - it sends a header that tells YT that you want English results, and that you're using a slightly outdated Safari, Chrome or Firefox Browser.
I would like to to have much more user agents to select from, but that's for another day. See for yourself what is hard-coded for now.

If you get 0 results, don't enter 's' to search again, or at least I think it's very unlikely to yield results the second time.
Exit the script, try again. With mytorsocks.sh you are choosing a new tor circuit each time, which is good.
Like I said, I made hundreds of searches this way today and had almost no duds.

Please help me troubleshoot again. Please look in /dev/shm/ for logs (this is a tmpfs on both debian and archlinux) and always include them when reporting here.

And btw, your /tmp (which is not a tmpfs on Debian) might now contain some old logs that you need to delete manually. And the same goes for /dev/shm if you never shut your machine down.

So, again, this is working better and better for me, and if it doesn't for you I'm willing to help.
Eventually I will also move this to its own repository...

Last edited by ohnonot (2020-10-22 19:27:38)


Give to COVAX! Here or here. (explanation)

Offline

#30 2020-10-22 18:58:28

damo
....moderator....
Registered: 2015-08-20
Posts: 6,663

Re: Search youtube without a browser

youtube-search is now working for me, including in my wrapper script. Nice one, thanks  kiss


Be Excellent to Each Other...
The Bunsenlabs Lithium Desktop » Here
FORUM RULES and posting guidelines «» Help page for forum post formatting
Artwork on DeviantArt  «» BunsenLabs on DeviantArt

Offline

#31 2020-10-24 17:06:06

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

^ Great!
I moved 3 scripts to their own repository now, which should be called "media-streaming-site-tools" but is called just "youtube-tools". In light of recent events, maybe that will turn out to be a mistake?
wink

Last edited by ohnonot (2021-05-27 17:32:19)


Give to COVAX! Here or here. (explanation)

Offline

#32 2020-10-25 18:32:40

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

Changes, mostly to mytorsocks.sh - multiple instances are now possible.
Please 'git pull'.


Give to COVAX! Here or here. (explanation)

Offline

#33 2020-12-19 19:34:32

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

The script youtube-search is a bash script again (*).
It now relies on youtube-dl, and jshon.

Previously the script was a python script NOT relying on youtube-dl, but I knew it would stop working evtl. because that's what youtube.com does: continuously introduce changes that make life hard for any kind of scraping.
The updated (provided by user teckk on LQ) python script is now youtube-search.py, but I'm sure it's just a matter of time before it stops working again.

Therefore it's best to rely on one central software project (namely youtube-dl) to deal with these challenges, and let my script be a wrapper.
It is slower now, esp. through tor.

mytorsocks.sh now has a retry with a new circuit option (-r) to make it easier to use when youtube sends a 429 (too many requests).
I am sorry for Martin who apparently had problems with tor & mytorsocks.sh, but that part always worked reliably for me.

(*) I have also simplified it a little. It does not take input from the clipboard or interactively anymore, only what is passed on the command line. OTOH it can pass the video page URL to your media player now, in addition to copying it to the clipboard (xclip).

Last edited by ohnonot (2020-12-28 08:06:23)


Give to COVAX! Here or here. (explanation)

Offline

#34 2020-12-24 11:58:38

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

Another update - youtube-search now asks for input ineractively again.

More Blah in the Readme, excerpt:

Scenario and possible workflow

Essentially I want a representation of the first page of results of a youtube search, without the obnoxious "I agree" popups that started overlaying everything recently.
That's what youtube-search is there for.
It will return familiar youtube page URLs (not the actual media URL) like "h.ttps://www.youtube.com/watch?v=XXXXXXXXXX" and copy those to the clipboard.
Then I use mpv-clip to grab that URL from the clipboard and make mpv+youtube-dl play it.

To avoid youtube kicking me out or delaying me for refusing to use their web pages in a browser, I run youtube-search through tor, with mytorsocks.sh, but stream the actual video through the "normal" WWW. That is why the above described clipboard solution works best for me.
mpv-clip has an option to replace youtube.com with a mirror site such as those listed at invidio.us.

Here are two commands I have bound to two hotkeys, one to search, one to play:

xterm -e 'mytorsocks.sh youtube-search -m 10 -- "$(xclip -o)"'
sh -c 'mpv-clip -y invidious.snopyta.org "$(xclip -o)"'

Give to COVAX! Here or here. (explanation)

Offline

#35 2020-12-30 16:24:02

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

Plenty of updates and improvements to both mytorsocks.sh and youtube-search.
Please refer to the README and to the -h output of the scripts.


Give to COVAX! Here or here. (explanation)

Offline

#36 2021-02-06 07:23:19

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

A few more changes to the mytorsocks.sh/youtube-search combo.


Give to COVAX! Here or here. (explanation)

Offline

#37 2021-05-03 05:09:47

ohnonot
...again
Registered: 2015-09-29
Posts: 5,534

Re: Search youtube without a browser

Somebody else wrote ytfzf meanwhile which is gathering a lot of attention (constant updates, 19 contributors already) - certainly more polished and featureful than mine - recommended.


Give to COVAX! Here or here. (explanation)

Offline

Board footer

Powered by FluxBB