btqueue

BTQueue 0.1.0 is available!

2006-02-27T19:54:00.000+07:00

I’m very exciting to announce the release of BTQueue 0.1.0. After very so long waiting for a year, I have finished BTQueue 0.1.0 and hopefully it works well without the classical bugs, e.g., 100% utilization and freezing.

This version is kind of beta release. You can leave comments and report bugs to help me fix them all. My purpose is to develop BitTorrent client for production run in data center and co-location.

This is the new beginning of BTQueue. By upgrading to 0.1.0, you are able to:

Utilize DHT network compatible to Bram Cohen’s client and BitComet
Query IP location from updated database plus AS number (see ip2cc)
Change client identifier to Azureus, Bram Cohen’s client, or BitComet

Most of all, lots of bugs should be fixed during the update of the core. Can’t wait? Let’s go to the download section right now. Any bug reports or comments are welcome at http://www.howforge.com/forum.

Technorati Tags: BTQueue, BitTorrent, Azureus, BitComet, DHT

HowForge.com

2006-01-06T21:40:00.000+07:00

I would like to introduce HowForge.com to all you here. It is my new home to share my thought and collection of know-how. For example, development, product, programming tutorial, software review, and many more.

The most recent post is about Grid Computing Deal of EMC and Acxiom and Sneak Preview SAP NetWeaver 04 SP15 Full Java Edition.

Another referal from unknown language

2005-12-21T22:24:00.000+07:00

Well, there is another referal at http://users.i.com.ua/~alexbes/a/btqueue.html I don't know what the language is but it should be Ukraine. Another one is http://robaco.mhx.com.br/blog/?q=node/92 in Brazil. Thanks LiveBlogStat.com to let me know these referals.

Referred by Slyck

2005-12-21T22:18:00.000+07:00

I have just found that BTQueue had been referred by Slyck at Slyck"s Guide to BitTorrent.

Technorati Tags: P2P, BitTorrent, BTQueue, Slyck, Guide

Reengineering of btqueue.sourceforge.net

2005-12-17T02:38:00.000+07:00

If you have ever noticed the changing of http://btqueue.sourceforge.net/, this is the first sign of BTQueue"s next generation. Actually, BTQueue 0.0.19 has been internally used by alpha testers for a year suddenly after 0.0.18 had been released. Unfortunately, frozen problem could not be permanently solved yet. I still don"t know what part caused the problem.

So, I decided to migrate core code, integrated to BitTornado, to original BitTorrent by Bram Cohen. I don’t mean BitTornado causes frozen problem. In fact, it would be the core code bridging BTQueue and BitTornado itself. Rewrite this code could lead to unstable but it must be worth to try. Nobody knows, the new code may be perfect for all of us.

This is still the main blog for BTQueue. The homepage at sourceforge will keep the documentation of BTQueue in blog format for easily publishing and editing online.

Progress

2005-12-10T18:57:00.000+07:00

Sorry all of you for so long waiting. Actually, I don"t have machine to run BTQueue anymore. That"s why development goes very slow. Furthermore, I was switched to do so boring work about Java.

Fortunately, I have backed to Python right now. I am looking for a machine to test BTQueue. The development should continue shortly.

BitTornado doesn"t support proxy

2005-12-10T18:51:00.000+07:00

I received a message in forum that requests for help about proxy setting in BTQueue. Unfortunately, BitTornado doesn"t use urllib2, instead it uses httplib for directly connecting to tracker. I am investigating to move to BitTorrent 4.2.1.

Sorry for long waiting...

2005-05-11T08:22:00.000+07:00

I apologize everyone waiting for the next version of BTQueue. I miss the plan to release it last two month ago oops. By the way, I will release it some day. Actually, I am finishing my thesis so this is a hard time in my life to write my own paper to publish in a journal. The development is slowly moving on but not freeze!

Plan to switch back to original BitTorrent

2005-03-10T08:48:00.000+07:00

I have tried to fix frozen problem in BTQueue since last two month but fail. It would be better to switch back to original BitTorrent by Bram. He has just released version 4.0.0.

PHPBTQueue

2005-01-23T00:43:00.000+07:00

I have found a project namely phpbtque at sourceforge. It is a Web interface to control BTQueue BitTorrent client. Written in PHP. Requires web server which understands PHP, and BTQueue BitTorrent client.

Fixed stop downloading bug in 0.0.18

2005-01-20T11:16:00.000+07:00

Finally, I have found the problem of BitTornado 0.3.9b. It will cancel requests that are not being responsed within snub time. After that it will choke itself. It is so hard to get unchoke from other peers at this point.

I just commented this code and BTQueue backs to the fastest client right now. I will release it as 0.0.19 very soon.

Tweaking your system to get the most out of BitTorrent

2005-01-19T10:38:00.000+07:00

Tweaking your system to get the most out of BitTorrent: BitTorrent built in the other half of trading you probably learned as a kid: tit-for-tat. That is, if you give me one piece, I'll give you one piece back. BitTorrent will give you a few pieces of the file for free to help you get started, but after that you need to start giving some pieces back if you want to keep downloading. If you don't share, eventually everyone else will stop talking to you. Just like when you were a kid with your toys on the playground. In fact, BitTorrent goes one step futher and actually starts to favor the people who share the most. This means that the more you upload, the faster you'll download.

New command

2005-01-18T10:07:00.000+07:00

Since I am encountering firewall problem which does not allow incoming connection, BTQueue looks so slow comparing to firewall which did not allow outgoing connection. To solve this problem, I guessed BT protocol implemented in BitTornado might have done something weird that make it act like that. So, I defined new command named "last_message" for verifying protocol interaction between BTQueue and other peers. In addition, you can set MAX_LAST_MESSAGE using gset command to specify number of messages kept in queue.

Serious bug in BitTornado 0.3.9b

2005-01-17T13:22:00.000+07:00

I found that BT 0.3.9b has a very serious bug. BT will not download as fast as it could through locally initiated connection. It means BT works best with incoming connection rather than outgoing connection made by itself. It turns out that clients behind firewall will be unable to download; however upload works fine here.

For the last year, I always ran BT behind firewall which blocks only outgoing connections to standard BT ports but incoming connections work fine. I found that incoming connection is the best condition to make BT as fast as it could even without outgoing connection at all.

Missing pyxml in 0.0.18 for Win32 has been fixed

2005-01-13T10:49:00.000+07:00

For those who downloaded BTQueue 0.0.18 for Win32, please download and install it again to fix missing pyxml package. It is necessary to upgrade to make crawler work properly. Sorry for that, it's my fault.

BTQueue 0.0.18 has been released!

2005-01-10T16:47:00.000+07:00

Eventually, BTQueue 0.0.18 has been released with lots of new features, improvements, and bugs fixed. In this version, xmlrpc is heavily used to fully support remote submission so that it is the first cluster-ready BitTorrent client. In addition, low disk space checking mechanism has been added to prevent consuming all disk in automatic crawler mode. Output table now is rendered dynamically to eliminate misalign column. Furthermore, you are allowed to specify favorite table formatting or define as alias. A most requested feature is file selection for multifile torrent. Actually, BitTornado provides this capability for long. However, BTQueue gives you more flexibility to generate priorities string for large number of files in a line. Peers command has also added to help you identify problem when no connection has been made in reasonable period. Don't waste your time. Try it right now! - fixed missing quote before saving xml - increased default rerequest interval to 20 minutes - upgraded to BitTornado 0.3.8 - fixed quoting unicode string - fixed incompatible of xmlrpc in 2.2 and 2.3 - reduced scrape thread to a single thread - changed to event scheduler instead of delay - checked for low disk space - rendered table dynamicly - added table header - fixed meta local file - fixed not creating first directory level in 2.3 - added download priorities in files - added --saveas and --priority to add command - displayed changed title if it is done automatically - fixed displaying usage by option -h - allowed to customize default command - removed unused codes - added returning id from do_add() - allowed adding metadata directly via xmlrpc - changed commandline to xmlrpc interface - added peers command to list of peers in connection queue - added priorities generation grammar

BTQueue 0.0.18

2005-01-04T10:13:00.000+07:00

I promise to release BTQueue 0.0.18 very soon. It might be this week or next week. Please wait a moment. I am adding the last feature to help all you to select files to download from a large set easier. I guess it is the easiest way in the world. Without this capability, file selector is totally useless!

Formatting table in BTQueue

2004-12-29T14:20:00.000+07:00

BTQueue 0.0.18 allows all you to specify your own table layout very easy. Basically, BTQueue has 3 tables for displaying brief information of torrents, connected peers, and all possible peers. The last table of all peers is introduced in version 0.0.18 to help you to verify the tracker action. This post aims to give you an overview to modify these format to meet your need. First, you have to know the basic. A table consists of a set of columns. What you can do is to rearrange column order, hide some of them, justify alignment, and cut too long text. Technically, a column can be represent by its alignment, optional maximum length, and its column name in form cd,ssss or c,ssss. c represents alignment where l and r stands for left and right, respectively. d is an optional number to limit maximum length of this column. In case of a column is longer than the limit, it will be cut off. The next field, ssss, is the column name.

Torrent Table

Name	Description
id	Identification Number
title	Torrent name. Usually, the first file or directory
progress	Progress in percentile
dlsize	Downloaded size
ulsize	Uploaded size
totalsize	Total size
eta	Estimated remaining time
dlspeed	Download rate
ulspeed	Upload rate
seeds	Number of seeders in bracket and number of connected seeders outside
peers	Number of leechers in bracket and number of connected leechers outside
btstatus	State
ratio	Shared ratio
activity	Current activity

Spew Table

Name	Description
ip	IP address
rport	Remote port
cc	Country code
direction	Connect direction
uprate	Upload rate
downlrate	Download rate
dtotal	Downloaded size
utotal	Uploaded size
completed	Progress
client	Client
netname	Network name

Peer Table

Name	Description
ip	IP address
rport	Remote port
cc	Country code
client	Client
netname	Network name

Using file selector to download individual files in a pack

2004-12-21T13:43:00.000+07:00

Oftenly, a torrent consists of lots of files but you just want to download only few of them. This feature is requested by many users especially low-bandwidth users who don't want to waste their bandwidth. According to BitTornado, it has supported this feature since last version. In BTQueue, it is enabled for easier use since 0.0.18. Actually, file selector in BTQueue is just to assign priority to each files in a torrent. There are 4 possible priorities from -1 until 2. -1 means to disable downloading this file. 0 is the highest priority while 2 is the lowest priority. In conclusion, BTQueue does not download files with priority -1 and it will download files with priority 0 prior to priority 2. There are 2 scenarios to specify priorities to each torrent. For newly added torrent, you can specify its priority directly as an option to add command. For example, a torrent consists of 2 files.

add -p -1,0 http://abc.com/abc.torrent

In other case, a torrent has been added and you want to specify its priorities. At this point, lset command can be used.

pause 1
lset 1 priorities -1,0
resume 1

ports/74883: Python 2.4 upgrade breaks port net/py-bittorrent [fix included] - news.gw.com

2004-12-20T10:04:00.000+07:00

ports/74883: Python 2.4 upgrade breaks port net/py-bittorrent [fix included] - news.gw.com: "After upgrading the port lang/python to version 2.4, the port net/py-bittorrent is broken. This appears to be an incompatibility in bittorrent's zurllib.py file; the patch below has also been sent to the bittorrent development list. This problem, and the same solution, appears to also apply to the port net/bittornado." I have applied this patch.

New features of BTQueue

2004-12-17T15:52:00.000+07:00

BTQueue 0.0.18 will be released very soon! I have fixed many bugs and make it restart itself if something goes wrong. In addition, some of you may want to download only some files. This feature has been implemented in BitTornado since last two version and now it is configurable in BTQueue now. Directory should be created correctly now. If you have a time to test it, please get it from CVS.

BTQFox

2004-12-09T14:50:00.000+07:00

BTQFox - The BTQueue Interface: I have just started trying to develop gui for BTQueue using XUL based on MAB. It is not usable yet but pretty cool!

Exeem

2004-12-06T01:29:00.000+07:00

Slyck Forums: "Exeem is a new file-sharing application being developed by the folks at SuprNova.org. Exeem is a decentralized BitTorrent network that basically makes everyone a Tracker. Individuals will share Torrents, and seed shared files to the network. At this time, details and the full potential of this project are being kept very quiet. However it appears this P2P application will completely replace SuprNova.org; no more web mirrors, no more bottle necks and no more slow downs. Exeem will marry the best features of a decentralized network, the easy searchability of an indexing server and the swarming powers of the BitTorrent network into one program. Currently, the network is in beta testing and already has 5,000 users (the beta testing is closed.) Once this program goes public, its potential is enormous."

Exeem

2004-12-06T01:18:00.000+07:00

Multiple commands in a single line

2004-12-02T16:16:00.000+07:00

I have just modified BTQueue to allow specifying multiple commands in a single line separated by semicolon (;). This feature is also available in alias so you can write simple procedure by yourself. Note that you have to quote semicolon if you want it being interpreted as normal character like before. For example, see below alias.

alias -s super_detail "detail %(arg1)s;spew %(arg1)s"

super_detail will call detail followed by spew.

Access Control List in BTQueue

2004-12-02T10:56:00.000+07:00

Since 0.0.15, BTQueue allows us to limit access of incoming and outgoing connection at real-time with powerful user interface at least for me. Its idea is based on access control list in routers, firewalls, and programs, e.g. apache. Basically, there are only 2 access control lists available in BTQueue; allow and deny. Each access control list is a list of individual IP, IP range by prefix, IP range by netmask, country code, or network name separated by comma. See some samples below.

TH,200.2.1.3,202.17.20.0/24,202.17.19.0/255.255.255.0,INET-TH

There are 2 special keywords to represent all IPs and none IP, ALL and NONE, respectively. What you can do to limit access from someone is to choose the order of allow and deny depending on your requirement. One may want to allow some IPs and deny other ones. One may want to deny some IPs and allow other ones. In order to specify ACL in BTQueue, you need to modify 3 options; order_acl, allow_acl, and deny_acl. By default, these options are defined as below.

order_acl = allow,deny
allow_acl = ALL
deny_acl = NONE

If you want to ban all connections from Singapore, these options should be set as below.

order_acl = deny,allow
allow_acl = ALL
deny_acl = SG

In order to allow local connections only in Singapore, the options should be set as below.

order_acl = allow,deny
allow_acl = SG
deny_acl = ALL

For information about country code and network name, you should consule spew command or ip command.

Switch back to ActiveState

2004-11-04T12:50:00.000+07:00

ActiveState - ActivePython free Python open source language distribution - Dynamic Tools for Dynamic Languages: "Current releases: ActivePython 2.4.0b1 (beta), ActivePython 2.3.4 ActivePython is ActiveState's quality-assured, ready-to-install distribution of Python, available for Linux, Solaris and Windows. As part of ActiveState's support for Python, we provide the ActivePython distribution free to the community. In addition to the core binary code, ActivePython currently includes commonly used external modules including expat for XML processing, zlib and bzip2 for data compression, the bsddb database library, Tcl/Tk 8.4.4; a suite of Windows tools developed by Mark Hammond, including the Pythonwin IDE, support for Python ASP, the PythonCOM system and more."

Oops! Official Python 2.3.4 doesn't come with pyxml

2004-11-01T12:16:00.000+07:00

I decided to use Python 2.3.4 for Windows from python team instead of Python 2.3.3 from ActiveState since 2.3.3 has a bug relating to bsddb. BTQueue 0.0.16 and later are packaged with python 2.3.4. Unfortunately, I forgot that pyxml was not installed by default. BTQueue 0.0.18 should be released very soon.

Features requested

2004-10-28T00:43:00.000+07:00

- Add some extension to incomplete files - Support wildcard in add command including recursive

Calculating ratio for very strick tracker

2004-10-25T01:57:00.000+07:00

In order to prevent cheating peer in closed tracker, there is a very easy solution to solve this problem. However, it might be too strick. Problem: Some clients do not even try to contact tracker but they are still able to leech using historical peers information. Solution: Count amount of download by size of the torrent immediately when download its metadata Advantage: Disable cheating of unreporting clients Disadvantage: Downloading only some files may not fair

Interesting tracker's promotions

2004-10-25T00:39:00.000+07:00

BT is very popular and some trackers do not allow non-member to leech from them to guarantee share ratio of the whole system. Below are interesting promotions for closed tracker.

Bonus credit - give extra upload by amount of snatched
Delay - do not allow low ratio users to leech new torrents
Golden time - turn off delay and count download only a half

In addition, many users have tried to cheat the counting system. It is mostly impossible to prevent cheating due to BT protocol. That means tracker must track not only share ratio, but also list of who want to leech and who finished each torrent for cross checking.

BitTornado 0.3.8 is now out

2004-10-23T08:22:00.000+07:00

BitTornado: "LATEST VERSION: 0.3.8 WHAT'S NEW: maintenance release works better with XP SP2 some bugs fixed GUI width problem known, will be fixed later randomized port selection" I have successfully merged changes made in this version into CVS.

Today bug fixed

2004-10-22T09:29:00.000+07:00

I have fixed missing quote to attribute of xml element before save. It is necessary to do so because I always unquote when read. Furthermore, I increase default rerequest interval to 20 minutes.

BTQueue 0.0.17 has been released

2004-10-20T10:04:00.000+07:00

Due to some minor bugs in 0.0.16, I decided to release this version to fix those bugs. Furthermore, sourceforge got some problems yesterday. No one can download. Link: BTQueue

Bugs in 0.0.16

2004-10-20T09:30:00.000+07:00

I've got bug reports of version 0.0.16. Most of them are unnecessary error message, e.g. scrape error due to network failed. Another bug relates to incompatibility of Popen4 on windows since I changed from popen to Popen4. I plan to release 0.0.17 which is a minor bug fix version very soon. Keep your eyes on btqueue!.

BTQueue 0.0.16 at freshmeat

2004-10-20T08:48:00.000+07:00

BTQueue 0.0.16 on the front page of freshmeat.

rerequest_interval is set too low

2004-10-19T01:22:00.000+07:00

SourceForge.net: Open Discussion: "Though I noticed some trackers banning me after a while. I figured out this was due to the setting rerequest_interval, which was set too low for most other clients and several trackers. I'd suggest setting this to something higher, like 1200 (20minutes), so users won't get banned without knowing why." Due to above suggestion, I will increase rerequest interval to 20 minutes in next version. However, you can set it to whatever you want to fit your need.

BTQueue 0.0.16 has been announced at the front page

2004-10-19T01:02:00.000+07:00

BTQueue 0.0.16 has been announced at the front page of sourceforge.net.

Daemon/Remote in BTQueue 0.0.16

2004-10-18T15:19:00.000+07:00

Since 0.0.16 introduces many new features, I am trying to explain how to use it efficiently. The most requested feature is to run it in daemon/service mode. Prior to this version, you (and me) have to run btqueue incorperately with screen to make it run likely in background. Actually, only some of you might want this feature and all of them run btqueue on his/her server placed at ISP. BTQueue perfectly fit the need of administrators for remotely leech/seed on share server or office. Today, you don't need screen anymore. All you want is just btqueue! However, old style btqueue is still preserved for short-term running. To run it as a daemon in the background, you have to specify daemon command.

$ btqueue.py daemon

After that, you will immediately return to prompt. Now you are ready to control btqueue anywhere by running command below:

$ btqueue.py remote

By default, btqueue.py will bind itself on localhost (127.0.0.1) to make it more secure. However, sometimes you might want to control it from anywhere. In this case, please change global setting xmlrpc_ip to 0.0.0.0 or another more specific ip. As you should already know, there are 2 ways to modify setting; 1) directly modify policy.conf and 2) call gset command in the btqueue's prompt.

>>> gset xmlrpc_ip 0.0.0.0

After modification, restarting is needed. In remote control mode, you can do whatever you used to do. Note that notifications such as start and terminate are disabled in this mode. Moreover, quiting in this mode will not terminate its daemon. In case of you want to shut down the daemon gracefully, you have to run kill command instead.

BTQueue 0.0.16 is out!

2004-10-18T14:58:00.000+07:00

I proundly release BTQueue 0.0.16 (build 240). This version is the major change to improve both functionality and comfortability. You can get it at sourceforget.