An important announcement from the founder of property-bee.com: The future of Property Bee is assured.

Bug #33: Support primelocation.com

An area to discuss the development and technical aspects of the toolbar

Bug #33: Support primelocation.com

Postby Beerhunter on Sun Oct 11, 2009 8:16 pm

Hopefully updated the dev database to support rpp007's work on primelocation.com;

New database tables + registration of site in the config;

Code: Select all
CREATE TABLE IF NOT EXISTS `primelocation_properties` (
  `property_id` int(10) unsigned NOT NULL,
  `pid_crc` int(11) NOT NULL,
  `pid` varchar(44) collate latin1_general_ci NOT NULL,
  `last_viewed` date NOT NULL,
  `user_id` int(10) unsigned NOT NULL,
  PRIMARY KEY  (`property_id`),
  KEY `pid_crc` (`pid_crc`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;


CREATE TABLE IF NOT EXISTS `primelocation_samples_old` (
  `sample_id` int(10) unsigned NOT NULL,
  `property_id` int(10) unsigned NOT NULL,
  `user_location_id` int(10) unsigned NOT NULL,
  `added_on` timestamp NOT NULL default CURRENT_TIMESTAMP,
  `sampled_on` timestamp NOT NULL default '0000-00-00 00:00:00',
  `data` varchar(1800) collate latin1_general_ci NOT NULL,
  PRIMARY KEY  (`sample_id`),
  KEY `twoway` (`property_id`,`added_on`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;

CREATE TABLE IF NOT EXISTS `primelocation_samples_recent` (
  `sample_id` int(10) unsigned NOT NULL,
  `property_id` int(10) unsigned NOT NULL,
  `user_location_id` int(10) unsigned NOT NULL,
  `added_on` timestamp NOT NULL default CURRENT_TIMESTAMP,
  `sampled_on` timestamp NOT NULL default '0000-00-00 00:00:00',
  `data` varchar(1800) collate latin1_general_ci NOT NULL,
  PRIMARY KEY  (`sample_id`),
  KEY `twoway` (`property_id`,`added_on`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;

INSERT INTO site_config ( site, last_recent_added_on, last_old_added_on, table_prefix ) VALUES ( 'pl', '2009-10-05 00:00:00', '2009-09-04 00:00:00', 'primelocation' )


Updated script to allow primelocation with the dev version of the toolbar;

Code: Select all
   $SUPPORTED = array(
      'dev'       => array('rm', 'df', 'pn', 'sspc', 'espc', 'gspc', 'pl' )
   );


rpp007 if you use the site abbreviation 'pl' in the rules, then you should be able to get the info into the servers db :o
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby rpp007 on Mon Oct 12, 2009 8:57 pm

Thanks for setting this up.

I've tried testing against the server DB but get an error. Looks to be standard 401 - toolbar not registered with property-bee.

Is there something I'm missing? The property refs are prefixed with pl, but I only have title, price - are the other items manadatory?
rpp007
 
Posts: 29
Joined: Fri Sep 25, 2009 8:40 pm

Re: Bug #33: Support primelocation.com

Postby Beerhunter on Mon Oct 12, 2009 9:48 pm

I see some rejected requests (between 20:50 and 21:08 today) for primelocation / rightmove, however the user id appears to be someone else (username begining with fo...)

Just to make sure it's not a bug in the backend....

If you have the SQLite Manager add-on installed, open up property-bee.sqlite and look at the preferences table. The row;
* uid I think has a value starting with 52rL
* lid I think has a value starting with nwdj

If thats the case it looks like the registration failed (for an unknown reason)

I think I've manually patched the dev database, so (assuming your uid/lid match what I think they are) it may work now.

Cheers
BH
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby rpp007 on Mon Oct 12, 2009 10:20 pm

I see some rejected requests (between 20:50 and 21:08 today) for primelocation / rightmove, however the user id appears to be someone else (username begining with fo...)


That was me, my bee username is 'fo...'.

I also forgot to set the 'pl' in the rules so there were a couple of requests with 'rm' that were for primelocation. I changed that and it still failed.

* uid I think has a value starting with 52rL
* lid I think has a value starting with nwdj

Yes, that's me

I just did a quick test and I think it's working now. Many thanks for sorting it.

Nearly done with primelocation. You've got to love templated design. All their pages for different types of property have the same HTML and have only a few key changes in the url. Nice. Fits very well with your rules based design. A few more nights on getting the agent/property details and then I'm done. Plus huge amounts of testing, of course ;-)

Do you still want it included in the next release? (2.0.6.0?) I'm about to commit my changes (to back them up - i'd hate to lose it all ) but thought I should stay off trunk if it's not going to go out. Let me know what you think.

cheers
rpp007
 
Posts: 29
Joined: Fri Sep 25, 2009 8:40 pm

Re: Bug #33: Support primelocation.com

Postby Beerhunter on Mon Oct 12, 2009 11:07 pm

Great... still not sure why it failed tho :(

I'm happy for you to comit to the trunk with your primelocation support.

When 2.0.6.0 is about to be released its just a case of whether to enable/disable primelocation support;
* in the toolbar (via the registerHostRules call)
* backend (by doing the above mod's on the live database/scripts).
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby rpp007 on Wed Oct 14, 2009 8:11 pm

I've committed the early changes for this. Not got as much done as I'd hoped, so still quite a bit to do.

Basic search pages should work for UK (sales/rental) and international.

To Do:
Tidy up rules.js -- some refactoring of common code
Fix some issues with parsing titles for Houses vs flats etc
Add rules for saved properties
Add regex validation
test
and more...
rpp007
 
Posts: 29
Joined: Fri Sep 25, 2009 8:40 pm

Re: Bug #33: Support primelocation.com

Postby rpp007 on Mon Oct 19, 2009 9:47 pm

ok, I've committed the latest work on this.

Unfortunately, I won't be able to pick up any more until the weekend as I'm away. However, if anyone would like to have a play they are most welcome ;-)

What needs to be done from code point of view:
  • tidy up the general format of the rules file
  • Review the xpaths for commonality and refactor. I've just put it in a 'working state'. e.g. _searchResultsPage and _searchResultsPageIntl are very similar.
  • Review the regex for the evaluting the prices (My regex isn't great and I've just copied the rightmove rules). On a side note I found a useful regex tool that didn't seem to like the format I copied from rightmove, but useful none the less.(http://gskinner.com/RegExr/)
  • Double check I've not used any layout xpaths. I've tried not to but there are some oddities like having to get the Agents details from an img alt tag, which I don't like, but can't see another way
  • jsdoc
  • Confirm I've actually grabbed the correct values for price, brief description, agent, etc, etc

What I've tested so far:
  • UK Buy - search and property detail
  • UK Rent - search and property detail
  • New Homes - search and property detail
  • International - search and property detail
  • MyPrimelocation - saved properties, recent properties. I've added all types of property I could see to saved properties and it seems ok.
  • Under offer and the rent equivalent status seems to come through ok

What needs to be done from a primelocation point of view:
  • I've not added the rules for Estate Agents. The url is of the form: http://www.primelocation.com/uk-estate- ... uk/y/sr/s/. You need to follow the links from Buy|Rent through to estate agent search; it's about 6 clicks.
  • similarly for commercial property and farms. Not sure how much we want to support this? However, looking at the page layout it may just be a case of adding the same rules to these URLs
  • I've not been able to find any delisted or missing properties so I don't know how it will handle those sorts of situation

Issues
  • I've noticed that text, which constitutes brief description can be different when displayed via a search page than on the detail page. (The search has it cut off). This causes changes in the brief description if you change between search view and detail view. How have you handled this in the past? Ignore the partial detail?
  • Some of the rental items contain dual currencies. I've seen the SQLlite DB seems to pick up the sterling amount so I assume that is ok
  • Some of the rental items contain a 'term' that the DB doesn't recognise. I've not dug through to see why. The properties are of the form 'Weekly rental of'

That's about it; I'm sure there are plenty of bugs/issues but I'll take a fresh look at the weekend and see what I can do.

cheers
rpp007
 
Posts: 29
Joined: Fri Sep 25, 2009 8:40 pm

Re: Bug #33: Support primelocation.com

Postby Beerhunter on Mon Oct 19, 2009 10:02 pm

rpp007 wrote:ok, I've committed the latest work on this.

Unfortunately, I won't be able to pick up any more until the weekend as I'm away. However, if anyone would like to have a play they are most welcome ;-)


I'll certainly give is try, and review those bits of the code you mention/answer the questions you posted before your back... and if I get time maybe commit/suggest a few changes too ;)

Not looked at primelocation that much before if at all (so have no idea how hard/easier it is to support) plus not looked at your primelocation rules yet... so very much starting from scratch for me too :shock:

Have a good time!
BH
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby rpp007 on Mon Oct 19, 2009 10:16 pm

Not looked at primelocation that much before if at all (so have no idea how hard/easier it is to support) plus not looked at your primelocation rules yet... so very much starting from scratch for me too


Welcome to our world ;-)

If you can cast your eye over it that would be very useful as there is lots to tidy up and I'd rather it 'conformed' as much as possible. I think it should be ok to maintain as they seem to have all their pages using a similar layout although I think they have a couple of bugs where they've dropped the odd div tag between pages that otherwise would be identical.

Have a good time!

I shall, off to Amsterdam for a wedding, not sure if I'll be producing my best code on my return :-)

cheers
rpp007
 
Posts: 29
Joined: Fri Sep 25, 2009 8:40 pm

Re: Bug #33: Support primelocation.com

Postby s-p on Tue Oct 20, 2009 10:04 am

Beerhunter wrote:
rpp007 wrote:ok, I've committed the latest work on this.

Unfortunately, I won't be able to pick up any more until the weekend as I'm away. However, if anyone would like to have a play they are most welcome ;-)


I'll get some tests run over the next couple of days and try in addition to adding some rules should I find time (unless BH get there first!) :)
Scott
s-p
 
Posts: 125
Joined: Mon Jun 09, 2008 10:20 pm

Re: Bug #33: Support primelocation.com

Postby s-p on Tue Oct 20, 2009 9:31 pm

rpp007 wrote:I shall, off to Amsterdam for a wedding, not sure if I'll be producing my best code on my return :-)

Hope you have a great time! I'll certainly not be jealous on my commute into work tomorrow morning. ;)


In regards to PrimeLocation support, I've added in a couple more URLs to support a few different search/detail pages.

Added support for:
Farms, Estates & Land for Sale [Search + Details pages]
Estate Agents [All Search types]


Test Results:
Issues:
  • Brief Description issue confirmed
  • Agent Address not being picked up on details page
  • Additional hyphen sometimes present on details page (issue with Prime Location site) - But could potentially filter out to prevent unnecessary title changes.

Random link failure: http://www.primelocation.com/new-homes/ ... MAB1709761
- I haven't looked into the cause of this issue.


I'll try and pick up some more stuff tomorrow if I can.
Scott
s-p
 
Posts: 125
Joined: Mon Jun 09, 2008 10:20 pm

Re: Bug #33: Support primelocation.com

Postby Beerhunter on Thu Oct 22, 2009 3:59 pm

Just going through the issues;

rpp007 wrote:
  • I've noticed that text, which constitutes brief description can be different when displayed via a search page than on the detail page. (The search has it cut off). This causes changes in the brief description if you change between search view and detail view. How have you handled this in the past? Ignore the partial detail?


Yup, there's 2 descriptions that the toolbar can log;

* "Brief Description" which is the text on a search page and maybe a truncated version of..
* "Detailed Description" which is the text on the property details page

I've committed a change, where "Brief Description" has been changed to "Detailed Description" in the relevent rules.

rpp007 wrote:
  • Some of the rental items contain dual currencies. I've seen the SQLlite DB seems to pick up the sterling amount so I assume that is ok


In the field Price.Value / Price.Currency fields?

Yup there's a limitation that restricts a price to one currency, so if there are multiple currencies, the first one (ie £ in this case) is used. Its not a major issue as Price.Value is only used (I think) to sort the sidebar by ascending/descending price.

BTW (whilst I think of it) there's also an issue with sites that allow you to change the currency, ie someone could search in euros, then someone else in £'s.. this would cause a change to be logged (even tho the actual price has not changed).

rpp007 wrote:
  • Some of the rental items contain a 'term' that the DB doesn't recognise. I've not dug through to see why. The properties are of the form 'Weekly rental of'


Ok, function PB_Service.getPriceID tries to split the price up into its numeric value/currency/term before inserting into the db.

I've commited a change so "Weekly rental of" / "Monthly rental of" will work correctly.

In the longerterm, maybe need to rethink this function, ie the rules for each site decide how to split up the price - however thats something to look at in the future.

rpp007 wrote:That's about it; I'm sure there are plenty of bugs/issues but I'll take a fresh look at the weekend and see what I can do.

cheers


I'll try and give the rules a bit of a test this evening to see if I can spot any issues.

Cheers
BH
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby s-p on Thu Oct 22, 2009 9:06 pm

Apologies for not getting around to testing any of these changes further. I've been struck down with the sniffles (Man Flu!) for the last couple nights. :(

I'll try and get testing/bug fixing as soon as I feel a bit better (hopefully tomorrow). :)
Scott
s-p
 
Posts: 125
Joined: Mon Jun 09, 2008 10:20 pm

Re: Bug #33: Support primelocation.com

Postby Beerhunter on Thu Oct 22, 2009 9:30 pm

s-p wrote:Apologies for not getting around to testing any of these changes further. I've been struck down with the sniffles (Man Flu!) for the last couple nights. :(

I'll try and get testing/bug fixing as soon as I feel a bit better (hopefully tomorrow). :)


Hope its not the cold (could of been flu as didn't eat for 2 days) going round here.. tho the initial affects dissapeared in a few days, took me 2 weeks to get rid of the damn thing :(

Hope you recover soon.

Meanwhile back to primelocation and 2.0.6.0, assuming we are going to beta this weekend, there are 3 options;

1) Not include primelocation
2) Unofficially support primelocation (so users can try but no guarentees about the history being kept)
3) Offically support primelocation

From a quick test, I think the code is good enough to include in 2.0.6.0, and so I think its down to a confidence test of whether 2) or 3) is the best route.

I'm looking for feedback from rpp007 and other developers that have looked at/tried primelocation.
User avatar
Beerhunter
Site Admin
 
Posts: 1788
Joined: Tue Jan 22, 2008 12:05 am

Re: Bug #33: Support primelocation.com

Postby s-p on Fri Oct 23, 2009 8:22 pm

I certainly think the code is good enough to go into this release. From my testing just now, I was unable to unearth any major issues that would prevent it from being available to the end user.

As for officially supporting it, that's dependant upon what criteria it needs to fulfil. I would be happy to support it in an official capacity, assuming we follow up support for those areas of the site not PB enabled in a future release:
  • Commercial - For Sale+Rental
  • Farms, Estates and Land - Rental

Given that these areas aren't exactly a priority, I don't think they should stop it's inclusion in the next release.

The only slight problem I could find with the current functionality, was the potential for one or two properties with a "Sold" status to be classified as "Available" by PB (i.e. http://primelocation.com/new-homes/details/id/NEWH999000784. Unfortunately this information is disclosed within the description section and not easily identifiable. In which case, it might as well be disregarded as an issue. :D

Good job rpp007!
Scott
s-p
 
Posts: 125
Joined: Mon Jun 09, 2008 10:20 pm

Next

Return to Development

Who is online

Users browsing this forum: No registered users and 8 guests

cron