Porthole 2011 GSOC project starting

Porthole has been chosen as one of this years GSOC projects. The project will encompass a complete re-structuring of porthole's backend so that it can make use of a new (soon to be stable) portage public API. Stable at this point means mostly a non changing interface for directly operating portage, getting information directly from portage while the backend code can change without fear of constant breakage for consumer apps like porthole.

The student doing the work is Cazou, the same person that worked to port kuroo to QT4 last year. This restructure is needed so that porthole can work even better with portage. It is also needed to be able to work properly with pkgcore which a backend for it will also be produced. That will give you more options as far as package managers are concerned.

More on this later...

All is not lost

Hey everyone. No I've not dropped off the edge of the earth.

I am gearing up for another season of mentoring for another gentoo gsoc project. This time it will be for porthole. This project is to finish to back-end re-structuring and create a new pkgcore back-end.

Other news, I am now the developer/maintainer for layman. I have implemented the new api I did last year. Extending and improving it some more. I have also moved the cli to use the new api. This has resulted in a 20-35% speed increase for the testing I've done. I've also started a porthole plug-in for it.

Even more porthole news:

Paul Varner has done some work and has helped me get porthole working in a prefix install. I am finishing up the changes, porting them from the dev branch into the master branch code. I'll push out a live ebuild and/or a -0.6.2_beta release hopefully soon. There has not been much in feature additions. mostly more code re-structure and cleanup so it should be easier to modify and maintain. some bugfixes...

Here is a screenshot of porthole running on fuzzyray's mac.

layman

Well, I've gotten everything rebased to layman again and merged into master. Along the way are several more fixes to the work I started in overlord. It is available in the layman-9999 ebuild in the main portage tree.

I'd appreciate all those that test it to make sure I haven't broken anything. I want to make a few more smaller changes to the code before making a layman-1.5 release. So test away, let me know of any problems or quirks that need tweaking.

With that part behind me , I've started to work on extending porthole's plugin system to be able to add custom views via plug-ins. My main goal is to get a layman plug-in working for porthole. I plan to use the existing porthole layout using the category listings for the installed, main remote list, add in lists. The package view for the short info listing and the summary (or possibly it's own tab) for the detailed info. With that part accomplished, I'll add the ability to add, delete, sync repositories.

Back to layman, I was thinking of adding a file in the same vein as portage and the main tree to record the last sync time. With that, i'll extend the layman cli to include an option to list the last sync time(s). It will also be available to api consumer apps like portholes layman plugin.

Layman related, I've been in touch with the dev that created a new git-2.eclass to see about extending it to be able to handle multiple listed EGIT_URI's with in it's declaration. That way if there are multiple protocols available or server locations it can be cloned from, it can cycle through them until one succeeds. just like layman does for adding an overlay to your system. That way if you are behind a firewall and the git:// protocol can not get through, it will try the next uri which may very well be an http:// protocol which can succeed. That should make live ebuilds more accessible to users.

meanwhile back to coding... :)

Getting Busy :)

Well, I know i haven't been very chatty here on my blog. Things have been getting quite busy.

I've been getting some work done on porthole:

1. I've split up the mainwindow code into smaller managable chunks that all stack together.
2. I've fixed the multi-selection for all commands like emerge, unmerge, update. Unfortunately I discovered a bad side effect of that. While selecting a bunch of dependencies I wanted to upgrade. I came across a pkg I emerged with --oneshot that I didn't want anymore. So clicked unmerge (old behavior would just unmerge the pkg selected), then continued selecting and then upgrade. To my surprise I discovered everything selected to upgrade failed due to wget failing. Then I discovered all the unmerge's which crippled my system. Fortunately I have --buildpkg enabled so was able to emerge a number of pkgs from the backups. My system is happily merging updates again. I still need to run revdep-rebuild after to catch any more missing deps from my unmerge stint.

I got to talking with Wizzleby (kportagetray developer) about the layman api I coded awhile back. That got us both going on some preliminary designs for collaborating on both a gtk and kde front-end to layman. Since I had not been getting any feedback about the new api from the main layman dev. I decided to start a fork to work on it more so that it can be released and ready for the guis front-ends. Well that lasted about a week, I made good progress on it, replaced the cli interface to use my new high level api. That resulted in a 20-35% speed increase. I am now the primary layman developer and maintainer. So now I have to port my changes back to layman's namespace. Anyway the new project's name is Overlord. I will keep it for the guis front-ends and possibly a few extras for layman. So give me some time and I'll have the changes in the Layman-9999 live ebuild ready for more testing.

I have been pushing to get the gentoo-guis mail list back in action again and discovered it hadn't been de-activated after all, but it was not listed either. So any of you interested in joining, it is gentoo-guis+subscribe@lists.gentoo.org. We welcome all

to join in.

We have started a new gentoo-guis overlay for our live ebuilds and development work. It is not yet listed in the main repositories list, but I do have a gentoo-guis repositories.xml file in the overlay you can download and configure layman to use to add it. Actually I made that easier to do in overlord. Those repo.xml definitions can be plugged in for automatic detection and inclusion. Yes, that will be ported to layman too :)

So, I am accepting volunteers for help with things on porthole, layman, overlord,...

:D

I've been a lazy blogger

For the few people following my blog, sorry I haven't posted anything for such a long time. The google summer of code project is well underway. My student is doing well.

I am mentoring a student for the kuroo project to revitalize it and bring it up to date with kde4 and integrate a new portage api into it so that it can keep current with changes in the vdb, etc.. It should bring a well liked portage gui back to life.

So a progress report:

kde4 porting is done and it has been changed to not rely on kde4 specificly but QT4. that should make it more portable to other desktops without the need of having more kde packages installed.

There is a kuroo4 live ebuild and a kuroo overlay, along with that are some other development pkgs. A layman-8888 ebuild that installs a branch I coded which creates a high level API to perform all layman actions. Unlike the command line interface this api is import-able into other programs, so running it in a terminal is not needed. In fact it will no be able to fully incorporate layman into any of the current/future portage gui's. To go along with that, Cazou has created c-layman which is a c interface lib that connects any program that can incorporate c lib's into it's sources. This is needed for kuroo as well as others to be able to interface with the api.

The new portage api, I have begun coding. I haven't gotten far with it yet, but I do need to get it done soon or Cazou will run out of work aside from chasing down bugs. Many of which are related to the old kuroo method of reading the portage metadata cache directly.

Porthole has been suffering lately as a result. I put out a -r2 to fix the missing import stopping it from saving config file changes to /etc/make.conf or /etc/portage/* files. And as a result of the way I have my system permissions, failed to get the patch correct, so it still doesn't work. I'll get it fixed soon, I promise.


I now have a working kvm setup using an init script from Brian Harring who is now a council member. With it I now have few kvm images that I can use for testing. One of those is a kde4 based desktop. Mostly for kuroo4 testing. I will be creating and using more images to better test porthole before releasing new versions so I can avoid the few blunders of the current ones.

My knee is doing a lot better and I have returned to work full time. I also am able to get some of the house/yard work done. We had been planning a number of house/yard reno's for many years and got some financing arranged last year. My knee injury postponed any progress on that front. But we have been getting started on a few things now. This weekend is a big cleanup push. I've been clearing some bushes, blackberries, etc. from along an old rotted fence, which is next on my list.

If there is anyone out there that wants to help incorporate any of the new things into porthole, give me a shout. I am going to be busy for the next year with work and house reno's so won't have the time I spent last fall/winter. There is lot's of new possibilities, improvements to porthole that need some coding.

That's all for now. I promise to keep this blog updated more frequently...

Busy, busy, busy,...

Wow, April has been a very busy month. I meant to keep you updated to the goings on, but didn't seem to get to it. My daughter's softball season has been up and going, not to mention all the other activities that she needs to be brought to, etc.... I have been starting back to work also on a part time basis, but my knee doesn't seem to be handling more than a 4 hour shift. I have also become a gentoo mentor for the Google Summer Of Code program. That has taken up quite a bit of time as well reading/evaluating proposals. I believe the announcement for the GSOC will come out later today (this is still my night before). I have also been contacting the other portage gui front-end developers to begin discussing the upcoming merge into portage of a public API. Coding wise I have not been able to accomplish much, mostly fix some bugs in gentoolkit, and do a little more work to portage's emaint that I've been re-working so that all its modules are available as import-able modules for use by portage, other maintenance tools, or even possible gui front-ends or plug-ins.

Oh, and some great news!!!! I have put in a stabilization request for porthole. So far it has a ppc stable keyword, approval for x86 and sparc to go stable, I just need to get the ebuild updated. So if you use porthole on any of the arch's that aren't marked stable, get after the arch team to stabilize it. If you are using it on an arch that is not yet listed in the ebuild, speak up, open a bug to get your arch added to the list.

Anyway, I have a busy day ahead for me tomorrow, so I should head off to bed. I'll be in touch about the gsoc project I'll be mentoring after the official announcement has been made.

status update

I've been a little busy lately and haven't had much time for a new post. I have the eclean re-write working good and have several unit tests for it, but to test it all would mean having 10 times the code just to run the tests on the real code. You will be able to try out the re-write in the gentookit-0.3.0_rc10 release coming out soon. For cleaning the packages directory it now relies on emaint from portage to fix the Packages index file. For now it runs it in a sub-process from eclean, but I hope to get some changes approved to emaint, so that eclean will be able to use it directly. It will then not disrupt the output from eclean like it will for now. Gentoolkit is now going to be python-3 capable and will require a minimum of python-2.6 to work correctly. Also, for those not following the gentoo-soc mail list, an idea for a network enabled eclean version was brought up that would be useful for central NFS servers serving the distfiles or packages directories. With that i have been able to deduce that it will not take much more work to add shared disfiles and/or packages ecleaning ability. So perhaps look for that feature in the near future, perhaps in gentoolkit-0.3.1.



It also looks like my knee injury is nearly healed up and I may be returning to work soon. A good thing about that is it might help to shed the 20+lbs of weight I put on since the injury.

I also need to code some additional modules for the new analyse utility I started, along with a major backend re-structuring for porthole. I've been also asked to clarify some of the analyse descriptions, abilities that I posted about earlier. So look for that this week. That's it for now.

Distractions

I've been distracted away form porthole code lately. Well not completely, I do have an ulterior motive. I am working on gentoolkit code again. Namely a modular rewrite of eclean.

eclean was an all in one python script that contained a number of unwieldy methods and functions which made it difficult to modify and nearly impossible to create test functions for. One of the primary reasons I started on it was to split the back-end functioning code from the output printing code similar to most of the newly redone equery modules. That way the code can be used by other applications. So, I began by separating the script into a python package containing 5 files and a nearly empty script that just loads and calls the main() .

Those 5 files being:

1. cli.py to handle all command line interface argument parsing and start the requested action.
2. search.py with all the search related methods and functions which determine the files to be cleaned.
3. exclude.py containing the exclusion file parsing and handling code.
4. output.py for the screen output and user interaction.
5. clean.py to handle the actual cleaning process.

The end result should be one that has a number of automatic tests run on it to check that any code modifications are performing correctly as well as code that will be easier to maintain.

It will also then be possible for the code to be used in other apps such as a planned plug-in module for porthole. :)

Translators Needed

I have porthole pretty much ready for another release. I have the translation files updated with any string changes and ready for a final edit.

1. Polish language is fully updated (just need to commit it). Thank you, Arkadiusz Kowalski.
2. German is nearly up to date, a few minor strings to check, Thank you, Timo Gurr.
3. French, is in need of checking and/or fixing.
4. Italian, is in need of updates.
5. Russian, is in need of updates.
6. Vietnamese, is in need of updates.

If you would like to see porthole translated into your language please leave a comment or email me. It is quite simple to do, just takes some time and knowledge of both English and the target language. There are tools like poedit to help.

And so Christmas came and the New Year begins

It's been the usual busy time of the year, with all the extra running around, preparing things. Getting the gifts for the people close to you. Trying to keep my wife from going overboard with the giving (again). It has been a good Christmas overall, lots of new toys, kids got some too (he he).

I had been hoping to get a release out for Christmas, but got a little too busy and lost concentration on what I was doing, so had a little break. The coding I was doing was centered more on su'ing porthole's processes that require root access while the majority of the code still runs as the user. I made a lot of progress, but came across a blocker in that gtk refused to do some things because of the detected su. It would run as all user or all root but not a mix. So I reverted most of the changes, (some were better even without su'ing). I believe I've come up with a better way without having to use policykit and the only python interface to it being through d-bus. Also due to the migration in python towards py-3k means using features of python-2.6 and greater. I am not about to do that for this release, but probably for the next major release of -0.7.0.

Porthole-0.7.0 I am hoping to have made all the changes needed to have multiple options for the backend package manager code it uses for data gathering as well as installing/syncing, etc..

So, I hope everyone had a wonderful Christmas and are starting the new year out right and happily so far.

gentoolkit fun

I've been having some fun working with djanderson on the gentoolkit equery code. We've made great speed improvements to the depends module for reverse dep searches. The real speed improvements were by djanderson's rewrite of the base search code after I had initially split it up into separate search and print modules. There have been 2 benefits so far

1. Is that consumer apps will be able to import the python modules for its own use without having to parse terminal output. Great for making a plugin for porthole (my original intent)
   
2. Is the major speed improvement as a result of the complete rewrite of all the dependency related code from the original package object into its own class. The speeds are now about 25% that of the original code. most equery depends package will now be around the 2 to 3 second mark instead of 8 to 12 seconds. Really long detailed searches are down from 2 - 3 minutes to around 2o seconds!

With the depends module nearing its completion, I'll get back to working on the belongs module. Maybe we'll be able to get more speed out of that one as well.

Back to coding

Current coding work I'm poking at:

Porthole

• I've just added syntax highlighting to the ebuild view courtesy of portato code. I would have done it earlier, but for trying to get the final version done with. I didn't want to add yet another change that might delay it's release due to more testing being needed...
• I've updated the file chooser widget used from the deprecated gtk.FileSelector to the new gtl.FileChooserDialog. It does also have file filtering options, and if I've done my testing correctly should be working properly everywhere it's used.
• I've gotten back to the dev branch that I'm using to split out all hard-coded emerge commands. The commands used to build the command lines used for emerging, etc. will come from config file(s). That will allow a user to customize the actual commands sent to the terminal. ie. use a custom wrapper script. It will also make it possible to use pkgcore commands for the merge. Once it is working, I'm sure some pkgcore devs will help to make a pkgcore backend for the information gathering as well.
• I've added a USE Flag tab to the package notebook. It is currently not selectable since I do not have any code for it yet. I plan to separate out the advanced emerge use flag portion of code into its own module so it can be used in both places, or anywhere else in the future.

Gentoolkit: I've begun poking around the gentoolkit code to see about using it for a plug-in module. I'm sad to say it was not designed to be used as an imported module. That said, I've seen on #gentoo-portage that djanderson has been working on the code. I've contacted him and have begun re-writing some of the code to separate out the data gathering and printing to the screen. This will improve the code base overall along with some major updates he is working on which will make the code py3k compliant as well as keep it up to date with current portage.
Together the changes should speed things up some for command line users. It will also make it possible for apps like porthole to import the gentoolkit code for its internal use and display of info such as reverse dependencies. (the first module I worked on splitting up)

Gpytage: A nice little gui utulity program for editing your /etc/portage/package.* files. I had helped Ken out a little in the past. I have started to poke at him some more to finish some updates he was doing so I can help debug it. I also hope to add some code to validate the entries and report any obsolete data & package entries. That code will come from another standalone utility script that I came across some time back. It will need some slight re-write to make into a library module and separate command line script. I am getting anxious waiting to make a plug-in module for porthole which will be linked in to the summary view and its menus. The existing version in the tree is somewhat outdated with portage, so will not be use-able. The new code will have a significantly different API.

IRC: I've created a #porthole support channel on freenode. feel free to drop by and chat. I can also be found on #gpytage, #gentoo-guis, #gentoo-portage, #gentoo-amd64, #gentoo-desktop, #pkgcore. I have been playing around with bots a little as well. You'll find a basic bot called portholes there. I intend to replace "portholes" with a different more powerful bot when I get the chance to get it set up.

First Rambling's

Welcome to my blog.

Well, this is something new to me. It was suggested that I get a blog to help promote Porthole and to get more feedback, etc. So here goes, It seems that like most things, they change and evolve. I remember a time when the forums were buzzing with activity. Some of that activity seems to have been replaced by blogs. I have been reading a few of the Gentoo dev's blogs and find them informative, interesting and can be a good source of knowledge. I hope to do well in that respect and not blab on about nonsense or get too carried away into details few people will care about.

Anyway, I am currently the only developer/maintainer for Porthole, It is a graphical gui front-end for Gentoo's package management system "portage". I was not the original creator(s), They were Daniel G. Taylor and Fredrik Arnerup. I started in fairly early in it's development cycle and have continued with it. There have been others that have come and contributed some good advancements and features over the years, but I am currently alone in it's development.

I welcome feedback from users, both good and not so good as to how they use porthole and things they might like added or changed. As in nearly all things, that does not mean it shall be done, but it does help evolve things for the better. I will try to get things done in a timely manner, but it is something I do in my spare time for fun, relaxation and good mental exercise. A person has to keep those synapses firing :)

I have recently released porthole-0.6.0 final after several years of trying to get it finished. This last big update from -0.5.x was a major one, porthole's code was getting too large and unwieldy in many ways and was in need of a major slice and dice. This has sped things up significantly and reduced it's memory usage. Along the way I have been able to add more features and improve functionality of others. There is still far more that I want to achieve with the code and user interface, but that discussion will have to wait for other blog entries.

Take care...