Blog posts about development

Return to latest posts

Hit the Highway with FixMyStreet

It’s something we’ve been wanting for a long time, and it’ll very soon be a reality: FixMyStreet reports will, where appropriate, be channeled to Highways England. Look out for this functionality in the coming week.

My way or the highway

Previously, if you reported a problem on one of the country’s motorways or major A roads, we had no way of identifying whether it was the responsibility of the government department rather than the council. We had to rely on whichever council the report fell within, and hope that they would forward it on.

But now, we can send reports off to just the right authority. What’s changed to make this improvement possible?

Well, FixMyStreet uses our MapIt software, which matches points (in this case, the pin you put in the map when you make a report) with the boundaries they fall within (mainly, until now, council boundaries). That’s how it knows which council to send your issue to, even if you have no idea yourself when you make the report.

Motorways and A roads have boundaries too, of course, but that data wasn’t previously available under an open licence that would allow us to use it on the site. That all changed with GOV.UK’s release of the Highways England Pavement Management System Network Layer — just what we needed!

So now, if you make a report that falls within a small distance from one of the relevant roads, FixMyStreet will use MapIt in combination with this data layer. You’ll see a message asking for confirmation that your report actually does pertain to the highway: where roads cross a motorway, for example, a pin could relate to the road on a bridge, or the motorway below.

Confirm either way and boom: off it goes to either Highways England or to the council, as appropriate.

So that’s a big thumbs up for open data: thanks, GOV.UK! It’s also a good example of how our commercial work, providing FixMyStreet Pro to councils as their default street reporting system, has a knock-on benefit across the open source FixMyStreet codebase that runs not only FixMyStreet.com, but sites run by other folk around the world.

As you may remember, we recently added red routes to Bromley for FixMyStreet Pro, and it was this bit of coding that paved the way for the highways work. We can only prioritise not-for-profit development if we have the funding for it; but being able to improve FixMyStreet for everyone on the back of work done for commercial clients is a win for everyone.

Or, as our developer Struan says, in a metaphor perhaps better suited to shipping routes than highways, “a rising tide raises all boats”.

 

Image: Alex Kalinin

 

 

Red routes on FixMyStreet

Our most recent improvement to FixMyStreet means that users in Bromley will experience some clever routing on their reports.

It’s something quite a few FixMyStreet users have requested, telling us that they’d reported a street issue in London, only to have a response from their authority to say that it was located on a ‘red route‘ — roads which are the responsibility of TfL rather than the council.

Of course, most councils have systems set up so that they can easily forward these misdirected reports to the right place, but all the same, it wasn’t ideal, and added another step into a reporting process we’ve always tried to keep as simple and quick as possible.

Thanks to some development for Bromley council, we’re now glad to say that within that borough, reports on red routes will automatically be forwarded to TfL, while other reports will be sent, as usual, to the relevant council department.

As a user, you don’t have to do a thing (although you can see this automated wizardry in action by watching changes in the text telling you where the report will be sent, as you click on the map in different places and select a different category – give it a go!).

Note that this functionality has not yet been extended to the FixMyStreet app; however in the meantime it will work if you visit fixmystreet.com via your mobile browser.

A new layer

As you’ll know if you’re a frequent FixMyStreet user, the site has always directed reports to the right UK council, based on the boundaries within which the pin is placed.

And equally, even within the same area it can discern that different categories of report (say, streetlights as opposed to parking) should be sent to whichever authority is responsible for them: that’s an essential in a country like the UK with its system of two-tier councils.

So this new innovation just meant adding in a map layer which gives the boundaries of the relevant roads that are designated red routes, then putting in extra code that saw anything within the roads’ boundaries as a new area, and TfL as the authority associated with road maintenance categories within that area.

FixMyStreet has always been flexible in this regard: you can swap map layers in or out as needed, leading to all sorts of possibilities. In a recent post, we showed how this approach has also averted one common time-waster for councils, and the same set-up is behind the display of council assets such as trees and streetlights that you’ll see for some areas on FixMyStreet.

The integration of red routes is available for any London Borough, so if you’re from a council that would like to add it in, get in touch. And to see all the new innovations we’re working on to make FixMyStreet Pro the most useful street reporting system it can be, make sure you’re subscribed to the Better Cities newsletter.

Image: Marc-Olivier Jodoin

What does backend integration actually mean?

mySociety developer Struan wrote a great update recently, describing everything we do when we integrate FixMyStreet Pro with a council’s system.

It was really for his colleagues to read, but it seemed like a shame to keep it only for our own consumption, because it’s a super-clear explanation, even for the less technically-inclined. So we thought we’d publish it here, as well. We hope you find it as useful as we did!

One of the benefits of FixMyStreet Pro’s Boulevard and Avenue plans is that, instead of sending emails to the council, we put reports directly into the council’s back end systems.

‘System’ can mean a lot of things, but for FixMyStreet we’re usually working to integrate with some sort of software that manages assets, often run by the council’s highways team. If you hear someone muttering about Confirm, Exor or WDM then we’re talking about one of these systems.

The most basic thing we can do when connecting FixMyStreet up to one of these systems is to take the information that a report-maker enters on FixMyStreet and put it into the equivalent fields in the council’s system. This is largely automating the process of cutting and pasting from an email that the council staff were probably doing before, and while that’s already a win for the council in terms of saving time and resources, we can usually do much better than this.

For example, we can pull a list of report categories out of the council’s system. This means that category changes don’t need to be made in both the backend system and FixMyStreet; they only need to happen in the backend system, and they’ll be automatically reflected on FixMyStreet without anyone having to remember to make the update on that side too.

This isn’t always practical, often because councils have many categories, not all of which they want to expose to the public. In those cases we still have to update the categories at our end. However, even doing this by hand in an integration allows us to add various bits of metadata to categories which are useful to the council. If we’ve automated this then all the metadata is pulled in as part of the automated process.

Depending on the council’s preferences/set-up, this metadata can include extra information to gather for certain categories, e.g. “how large is the pothole?” or information such as streetlight numbers associated with the category. These can help the council prioritise reports and also reduce the need to go back to the user to ask for further information.

The other thing integrations allow us to do, is both send — and more importantly for the council— fetch updates on reports. Not only does this mean that the council only needs to update the problem in their internal system; it enables better reporting on the current problem state.

While a member of the public can only mark a problem as fixed, a greater range of states are available to councils (Investigating, Work Scheduled and so on) which helps make it a bit clearer what’s happening with a report. Councils often have a fairly complicated internal workflow so another step in the integration can be mapping their internal status codes to those displayed on FixMyStreet.

From a public perspective, this also means that updates to a problem get to the council (we don’t email updates on reports; they’ve historically been presented as a way for FixMyStreet users to discuss an issue rather than to chase an issue or note a change in its status). This is also good for the council as it means if a member of the public isn’t happy with their response then they find out.

The gritty technical details

FixMyStreet’s native way to communicate with outside systems is Open311, an open standard for problem reporting.

Unfortunately, most council systems don’t speak Open311, so we need to write code that sends information in whatever format they use. We used to do this by adding code directly into FixMyStreet, which wasn’t too bad as the problem sending code is pretty modular and supports plug-ins for the actual sending process.

Once we started adding more integrations we wanted to move away from this approach, largely to avoid clogging up the main FixMyStreet codebase. As Open Source code, FixMyStreet has been deployed in many countries around the world, and it is unlikely that people running their own sites abroad would need this part of the code, and certainly not the whole range of different systems we were covering.

So now, most of our integrations communicate through a proxy that accepts Open311 requests, transforms them into requests to the council’s backend system and then transforms the responses back into Open311. The proxy is an evolution of the code our ex-colleague Hakim wrote some time ago for a previous integration.

Open311 is a pretty flexible standard, so there’s lots of room to pass custom information about, and that means we can hide quite a lot of the complexity in the proxy and never have to touch FixMyStreet’s code.

For councils where we can’t automatically fetch the list of categories, all the configuration is included in the proxy, making this invisible to FixMyStreet.

As more and more councils come on board, the process of adding integrations to the proxy will become easier. For example, now that we’ve set up a few Confirm integrations, any new clients using Confirm will require less code-writing and it’ll just be a matter of configuring mappings and categories.

And even if we’re integrating with a system we haven’t come across before, we now have a standard pattern of work, meaning that more of the code is in the specific implementation, and less in the set-up. That makes for a quicker, easier implementation all round.

Want to know more? Drop us a line and we’ll happily answer all your questions.

Image: Randall Bruder

Seeing spots with FixMyStreet Pro

If you visit FixMyStreet and suddenly start seeing spots, don’t rush to your optician: it’s just another feature to help you, and the council, when you make a report.

In our last two blog posts we announced Buckinghamshire and Bath & NE Somerset councils’ adoption of FixMyStreet Pro, and looked at how this integrated with existing council software. It’s the latter which has brought on this sudden rash.

At the moment, you’ll only see such dots in areas where the council has adopted FixMyStreet Pro, and gone for the ‘asset locations’ option: take a look at the Bath & NE Somerset installation to see them in action.

What is an asset?

mySociety developer Struan explains all.

Councils refer to ‘assets’; in layman’s language these are things like roads, streetlights, grit bins, dog poo bins and trees. These assets are normally stored in an asset management system that tracks problems, and once hooked up, FixMyStreet Pro can deposit users’ reports directly into that system.

Most asset management systems will have an entry for each asset and probably some location data for them too. This means that we can plot them on a map, and we can also include details about the asset.

When you make a report, for example a broken streetlight, you’ll be able to quickly and easily specify that precise light on the map, making things a faster for you. And there’s no need for the average citizen to ever know this, but we can then include the council’s internal ID for the streetlight in the report, which then also speeds things up for the council.

Map layers

So, how do we get these assets on to the map? Here’s the technical part:

The council will either have a map server with a set of asset layers on it that we can use, or they’ll provide us with files containing details of the assets and we’ll host them on our own map server.

The map server then lets you ask for all the streetlights in an area and sends back some XML with a location for each streetlight and any associated data, such as the lamppost number. Each collection of objects is called a layer, mostly because that’s how mapping software uses them. It has a layer for the map and then adds any other features on top of this in layers.

Will these dots clutter up the map for users who are trying to make a report about something else?

Not at all.

With a bit of configuration in FixMyStreet, we associate report categories with asset layers so we only show the assets on the map when the relevant category is selected.

We can also snap problem reports to any nearby asset which is handy for things like street lights as it doesn’t make sense to send a report about a broken street light with no associated light.

Watch this space

And what’s coming up?

We’re working to add data from roadworks.org, so that when a user clicks on a road we’ll be able to tell them if roadworks are happening in the near future, which might have a bearing on whether they want to report the problem — for example there’s no point in reporting a pothole if the whole road is due to be resurfaced the next week.

Then we’ll also be looking at roads overseen by TfL. The issue with these is that while they are always within a council area, the council doesn’t have the responsibility of maintaining them, so we want to change where the report is going rather than just adding in more data. There’s also the added complication of things like “what if the issue is being reported on a council-maintained bridge that goes over a TFL road”.

There’s always something to keep the FixMyStreet developers busy… we’ll make sure we keep you updated as these new innovations are added.

From a council and interested in knowing more? Get in touch.

Confirming the benefits: how FixMyStreet Pro integrates with the Confirm Asset Management System

We often talk about how FixMyStreet Pro can integrate directly with council’s existing systems, and how doing so can help councils be more efficient — but what exactly does that mean in practice?

Let’s take a look at our two most recent FixMyStreet Pro installations. Both B&NES and Buckinghamshire councils use the same asset management system, Confirm, and it gives us a great example of how FixMyStreet Pro’s ability to ‘communicate’ with such systems will make everything a whole lot easier for residents and for council staff, even with two very different types of local authority.

Saving time and effort

FixMyStreet has always provided the resident with an easy interface through which to file a street report. For many councils, however, such reports arrive in an email inbox and then have to be forwarded to the right location or typed into the council’s CRM, all adding to the sum total of time and effort dedicated to each report.

Now, using the Confirm API, Bucks and B&NES councils can access and work on FixMyStreet reports through Confirm’s standard ‘inspector module’, removing any need for this extra step.

Two-way information

Optionally, the information flow can go both ways, and indeed this is the case for both B&NES and Buckinghamshire councils. What this means is that for example, when an issue has been inspected and council staff change its status (perhaps from ‘report received’ to ‘repair underway’), this status change will be passed back to FixMyStreet, automatically syncing with the site, and notifying the report-maker with the update — again removing another mundane task from customer services staff.

If a highways inspector should come across a new issue while they are out and about on their rounds, they can raise an issue in Confirm just as they always would have. But now, that will also create a report on FixMyStreet which residents can view, keeping everyone up to date and ensuring that reports aren’t made about issues that the council already know about.

Canned responses

FixMyStreet Pro also allows for council administrators to create template responses — an invaluable timesaver when responding to one of the more common situations such as “issue identified and prioritised” or “issue now fixed and closed”. While Confirm also has its own template responses, FixMyStreet Pro offers more flexibility, as the same template can be reused across multiple report categories and status types. Buckinghamshire really saw the benefit of this: they were able to reduce the number of templates in use from around 450 to 46.

Mapping assets

Assets such as streetlights, grit bins and gullies can be pulled through from Confirm and overlaid on the map. This makes it significantly easier for both residents and staff to locate and report issue, speeding up the issue resolution time — we’ll be delving more deeply into this in our next blog post, with a few more technical details for those who are interested.

Image: Highways England (CC by/2.0)

We’ve upped the security on FixMyStreet

We’ve recently introduced some stronger privacy and security measures on FixMyStreet, to make things safer for everyone. They also have some nice knock-on effects that help you with moderation.

Privacy

If you’re a FixMyStreet Admin, you can now:

  • Make a user anonymous across the site, so even if they’ve made multiple reports, their name won’t show on any of them on the live web pages. Removing users’ names is a frequent request, especially from those who may have strong personal reasons not to be identified online. Users already had the ability to anonymise their reports singly or in bulk themselves, but sometimes it’s easier to do it for them, particularly if they are distressed when making the request.
  • Remove a user’s account details entirely An important point in the forthcoming GDPR regulations is that we all have the right to request the removal of our personal data from databases. In this case, the user’s reports and updates remain, but not only is the name removed from public webpages as per the point above; their email address, phone number and any other personal data are scrubbed from our own servers, too, leaving no record.
  • Hide all a user’s reports/updates from the live site. In the event that you discover a large quantity of, say, abusive reports from the same person, you can now remove them all from the online environment at a single stroke.

Security

Security for users was already very good, but with the following improvements it can now be considered excellent!

  • All passwords are now checked against a list of the 577,000 most common choices, and any that appear in this list are not allowed.
  • Passwords must now also be of a minimum length.
  • If you change your password, you have to input the previous one in order to authorise the change. Those who haven’t previously used a password (since it is possible to make a report without creating an account), will receive a confirmation email to ensure the request has come from the email address given.
  • FixMyStreet passwords are hashed with an algorithm called bcrypt, which has a built in ‘work factor’ that can be increased as computers get faster. We’ve bumped this up.
  • Admins can now log a user out of all their sessions. This could be useful for example in the case of a user who has logged in via a public computer and is concerned that others may be able to access their account; or for staff admin who share devices.

Still got any questions about privacy or security? Drop us a line and we’ll be glad to answer them.


Image: Timothy Muza (Unsplash)

Multi-selectin’ fun on FixMyStreet

As a user or a council, it’s quite possible that you’re already enjoying one of the usability improvements that FixMyStreet version 2.0 has brought — but, as it’s a fairly subtle change, perhaps without actually even noticing it.

multi-select

 

In these days of eBay and department store shopping, we’re all quite used to refining results through the use of multiple checkboxes.

But for FixMyStreet, we hadn’t given much thought to letting you filter reports by more than one dimension, until Oxfordshire County Council suggested that it would be a useful feature.

For quite some time, you’d been able to filter by category and status (“Show me all pothole reports” or “Show me all ‘unfixed’ reports”), but this new functionality is more flexible.

You can now select multiple categories and multiple statuses simultaneously (“show me all pothole and graffiti reports that are unfixed or in progress”) — and all through the power of tickboxes.

If you’re a non-technical person, that’s all you need to know: just enjoy the additional flexibility next time you visit FixMyStreet. But if you are a coder, you might like to read more about how we achieved this feature: for you, Matthew has written about it over on the FixMyStreet Platform blog.

If you’d like to know more about all the features we’ve recently introduced to FixMyStreet, why not join one of our regular Friday webinars?

Peak performance

At mySociety we believe in an open, inclusive web and such we try to build web apps that are accessible in the broadest sense. So while we do care deeply about things like WAI and the Equality Act this post isn’t about that — this is about making a site that works if you have a weak connection or an ageing device. I’m talking about performance.

Graph showing total transfer data for mobile webpages in last year.Now while it isn’t a great metric to track, the fact that the average size of a web page is now over three megabytes (and pages served for mobile devices reaching an average of 2.9mb!) demonstrates that this is an age of bloat that assumes good broadband or 4G connectivity and we don’t think that’s right.

As an example here are some numbers about the FixMyStreet site as it displays on mobile after some recent improvements.

To load a working and styled front page on your phone takes around 9KB of HTML/inlined CSS/inlined images (that isn’t a typo – nine kilobytes). How do we pull that off? Well, the site logo and menu are both inlined so we don’t have to wait for them to load, as is the CSS needed to show the top part of the front page. 5KB of JavaScript is loaded (which amongst other things enables the geolocation) and in the background an additional massive 14 kilobytes of CSS (the main mobile stylesheet) and the remaining 20 kilobytes of images (the example report photos and footer links) are being pulled in. The page also uses prefetch to start fetching the remaining JavaScript while the user is entering a postcode or address to actually get started on FMS.

On a desktop there’s a little bit more to add to the mix (more like 66KB of images, 19KB of CSS, plus a webfont taking 77KB) but it’s still lightning quick.
The team haven’t reinvented the wheel to achieve this – they’ve just been ruthless and absolutely focused on only using the minimum amount of code to meet the user need. When the FixMyStreet site is deployed, the JavaScript and CSS is automatically minimised, and at that point we run penthouse> to work out the critical CSS to be inlined on the front page. And whilst our main JavaScript does use jQuery, we dropped it from the front page to save yet more up-front time (jQuery is far larger alone than our current front page).

If you are interested in more details of how this was achieved, here’s a post Matthew prepared earlier on many of the same techniques, which he used on his own project traintimes.org.uk.

There are of course still improvements to be made – I imagine many front page viewers of FixMyStreet never need or want to scroll down as far as the images in the footer, so ideally we wouldn’t load them unless they do. Due to Windows Lumia users, which we support for a specific client use case, we’re using Appcache for offline support, but adding some form of more modern service worker would also be nice. And most of this work is for the front page (though it helped other pages too); our main JavaScript could be split up more than it is. It’s a continual process, but here is a good place to pause.

Photo by Braden Collum on Unsplash

FixMyStreet reaches Borsetshire

What would Eddie Grundy do if he came across a pothole? And how would Linda Snell deal with flytipping on the site of the Ambridge village fete?

Fortunately, these fictional characters now enjoy the same access to FixMyStreet as the rest of us, thanks to the new demo site we’ve built.

The thinking behind it is not, of course, to gather reports from an entirely fictional world. We’re not that mad.

Rather, we needed a sandbox interface where we could show councils exactly how FixMyStreet works, and allow them to play about with both the customer end and the admin side, all without causing any major repercussions to the running of the standard site. Enter FixMyStreet Borsetshire.

Prospective buyers of the system from local councils can experience the various levels of administration that the back-end allows. Just log in with the credentials seen on this page and see exactly how reports can be shortlisted, actioned, or moderated.

So, we’re expecting reports of pigs on the loose, flooded culverts and perhaps even a flying flapjack. But if you’re hoping to find out the precise location of Ambridge, unfortunately you’ll be disappointed: the map is actually centred around Chipping Sodbury, far from the village’s supposed Midlands locale.

Keen to know more? Join one of our regular Friday webinars for the full tour.

Image: Martin Pettitt (CC by/2.0)

How councils get extra benefits, at no extra cost

We’re always making improvements to FixMyStreet. And the great thing is that, as a council that’s installed FixMyStreet Pro as your fault-reporting system, you get all those improvements for free.

With most SAAS providers, there’s no real incentive to keep refining a product once it’s sold — but FixMyStreet is a bit different. The same software that underpins FixMyStreet Pro also drives our nationwide site for the UK, FixMyStreet.com, and a number of FixMyStreet sites run by other groups across the world. When we make an improvement to the codebase, that’s rolled out so that everyone can benefit from it.

The latest updates can always be seen on the FixMyStreet Platform blog, where, as you’ll see, some recent additions include the potential to update reports by text rather than email, and some performance updates as well as bug fixes.

Because FixMyStreet is in a continual state of development, we are always happy to hear from councils about the features they’d like to see. Just let your account manager know, or, if you’re familiar with the site GitHub, you can raise a ticket directly on the public FixMyStreet repo.

And when we make announcements on the blog, if you see anything you’d like to carry across to your own installation, drop us a line and we can discuss how best to make it happen.


Image: Štefan Štefančík (Unsplash)

FixMyStreet Pro blog

FixMyStreet Pro is the street & environment reporting service that integrates with any council system.

Join our newsletter

Be the first to know about FixMyStreet-related launches and events, surprising stats, and digital breakthroughs.

Join one of our regular webinars

Show webinar schedule

Schedule your one-to-one demo

Request a callback

See FixMyStreet Pro for yourself

Try our live demo