Blog posts about development

Return to latest posts

What does backend integration actually mean?

mySociety developer Struan wrote a great update recently, describing everything we do when we integrate FixMyStreet Pro with a council’s system.

It was really for his colleagues to read, but it seemed like a shame to keep it only for our own consumption, because it’s a super-clear explanation, even for the less technically-inclined. So we thought we’d publish it here, as well. We hope you find it as useful as we did!

One of the benefits of FixMyStreet Pro’s Boulevard and Avenue plans is that, instead of sending emails to the council, we put reports directly into the council’s back end systems.

‘System’ can mean a lot of things, but for FixMyStreet we’re usually working to integrate with some sort of software that manages assets, often run by the council’s highways team. If you hear someone muttering about Confirm, Exor or WDM then we’re talking about one of these systems.

The most basic thing we can do when connecting FixMyStreet up to one of these systems is to take the information that a report-maker enters on FixMyStreet and put it into the equivalent fields in the council’s system. This is largely automating the process of cutting and pasting from an email that the council staff were probably doing before, and while that’s already a win for the council in terms of saving time and resources, we can usually do much better than this.

For example, we can pull a list of report categories out of the council’s system. This means that category changes don’t need to be made in both the backend system and FixMyStreet; they only need to happen in the backend system, and they’ll be automatically reflected on FixMyStreet without anyone having to remember to make the update on that side too.

This isn’t always practical, often because councils have many categories, not all of which they want to expose to the public. In those cases we still have to update the categories at our end. However, even doing this by hand in an integration allows us to add various bits of metadata to categories which are useful to the council. If we’ve automated this then all the metadata is pulled in as part of the automated process.

Depending on the council’s preferences/set-up, this metadata can include extra information to gather for certain categories, e.g. “how large is the pothole?” or information such as streetlight numbers associated with the category. These can help the council prioritise reports and also reduce the need to go back to the user to ask for further information.

The other thing integrations allow us to do, is both send — and more importantly for the council— fetch updates on reports. Not only does this mean that the council only needs to update the problem in their internal system; it enables better reporting on the current problem state.

While a member of the public can only mark a problem as fixed, a greater range of states are available to councils (Investigating, Work Scheduled and so on) which helps make it a bit clearer what’s happening with a report. Councils often have a fairly complicated internal workflow so another step in the integration can be mapping their internal status codes to those displayed on FixMyStreet.

From a public perspective, this also means that updates to a problem get to the council (we don’t email updates on reports; they’ve historically been presented as a way for FixMyStreet users to discuss an issue rather than to chase an issue or note a change in its status). This is also good for the council as it means if a member of the public isn’t happy with their response then they find out.

The gritty technical details

FixMyStreet’s native way to communicate with outside systems is Open311, an open standard for problem reporting.

Unfortunately, most council systems don’t speak Open311, so we need to write code that sends information in whatever format they use. We used to do this by adding code directly into FixMyStreet, which wasn’t too bad as the problem sending code is pretty modular and supports plug-ins for the actual sending process.

Once we started adding more integrations we wanted to move away from this approach, largely to avoid clogging up the main FixMyStreet codebase. As Open Source code, FixMyStreet has been deployed in many countries around the world, and it is unlikely that people running their own sites abroad would need this part of the code, and certainly not the whole range of different systems we were covering.

So now, most of our integrations communicate through a proxy that accepts Open311 requests, transforms them into requests to the council’s backend system and then transforms the responses back into Open311. The proxy is an evolution of the code our ex-colleague Hakim wrote some time ago for a previous integration.

Open311 is a pretty flexible standard, so there’s lots of room to pass custom information about, and that means we can hide quite a lot of the complexity in the proxy and never have to touch FixMyStreet’s code.

For councils where we can’t automatically fetch the list of categories, all the configuration is included in the proxy, making this invisible to FixMyStreet.

As more and more councils come on board, the process of adding integrations to the proxy will become easier. For example, now that we’ve set up a few Confirm integrations, any new clients using Confirm will require less code-writing and it’ll just be a matter of configuring mappings and categories.

And even if we’re integrating with a system we haven’t come across before, we now have a standard pattern of work, meaning that more of the code is in the specific implementation, and less in the set-up. That makes for a quicker, easier implementation all round.

Want to know more? Drop us a line and we’ll happily answer all your questions.

Image: Randall Bruder

Seeing spots with FixMyStreet Pro

If you visit FixMyStreet and suddenly start seeing spots, don’t rush to your optician: it’s just another feature to help you, and the council, when you make a report.

In our last two blog posts we announced Buckinghamshire and Bath & NE Somerset councils’ adoption of FixMyStreet Pro, and looked at how this integrated with existing council software. It’s the latter which has brought on this sudden rash.

At the moment, you’ll only see such dots in areas where the council has adopted FixMyStreet Pro, and gone for the ‘asset locations’ option: take a look at the Bath & NE Somerset installation to see them in action.

What is an asset?

mySociety developer Struan explains all.

Councils refer to ‘assets’; in layman’s language these are things like roads, streetlights, grit bins, dog poo bins and trees. These assets are normally stored in an asset management system that tracks problems, and once hooked up, FixMyStreet Pro can deposit users’ reports directly into that system.

Most asset management systems will have an entry for each asset and probably some location data for them too. This means that we can plot them on a map, and we can also include details about the asset.

When you make a report, for example a broken streetlight, you’ll be able to quickly and easily specify that precise light on the map, making things a faster for you. And there’s no need for the average citizen to ever know this, but we can then include the council’s internal ID for the streetlight in the report, which then also speeds things up for the council.

Map layers

So, how do we get these assets on to the map? Here’s the technical part:

The council will either have a map server with a set of asset layers on it that we can use, or they’ll provide us with files containing details of the assets and we’ll host them on our own map server.

The map server then lets you ask for all the streetlights in an area and sends back some XML with a location for each streetlight and any associated data, such as the lamppost number. Each collection of objects is called a layer, mostly because that’s how mapping software uses them. It has a layer for the map and then adds any other features on top of this in layers.

Will these dots clutter up the map for users who are trying to make a report about something else?

Not at all.

With a bit of configuration in FixMyStreet, we associate report categories with asset layers so we only show the assets on the map when the relevant category is selected.

We can also snap problem reports to any nearby asset which is handy for things like street lights as it doesn’t make sense to send a report about a broken street light with no associated light.

Watch this space

And what’s coming up?

We’re working to add data from roadworks.org, so that when a user clicks on a road we’ll be able to tell them if roadworks are happening in the near future, which might have a bearing on whether they want to report the problem — for example there’s no point in reporting a pothole if the whole road is due to be resurfaced the next week.

Then we’ll also be looking at roads overseen by TfL. The issue with these is that while they are always within a council area, the council doesn’t have the responsibility of maintaining them, so we want to change where the report is going rather than just adding in more data. There’s also the added complication of things like “what if the issue is being reported on a council-maintained bridge that goes over a TFL road”.

There’s always something to keep the FixMyStreet developers busy… we’ll make sure we keep you updated as these new innovations are added.

From a council and interested in knowing more? Get in touch.

Confirming the benefits: how FixMyStreet Pro integrates with the Confirm Asset Management System

We often talk about how FixMyStreet Pro can integrate directly with council’s existing systems, and how doing so can help councils be more efficient — but what exactly does that mean in practice?

Let’s take a look at our two most recent FixMyStreet Pro installations. Both B&NES and Buckinghamshire councils use the same asset management system, Confirm, and it gives us a great example of how FixMyStreet Pro’s ability to ‘communicate’ with such systems will make everything a whole lot easier for residents and for council staff, even with two very different types of local authority.

Saving time and effort

FixMyStreet has always provided the resident with an easy interface through which to file a street report. For many councils, however, such reports arrive in an email inbox and then have to be forwarded to the right location or typed into the council’s CRM, all adding to the sum total of time and effort dedicated to each report.

Now, using the Confirm API, Bucks and B&NES councils can access and work on FixMyStreet reports through Confirm’s standard ‘inspector module’, removing any need for this extra step.

Two-way information

Optionally, the information flow can go both ways, and indeed this is the case for both B&NES and Buckinghamshire councils. What this means is that for example, when an issue has been inspected and council staff change its status (perhaps from ‘report received’ to ‘repair underway’), this status change will be passed back to FixMyStreet, automatically syncing with the site, and notifying the report-maker with the update — again removing another mundane task from customer services staff.

If a highways inspector should come across a new issue while they are out and about on their rounds, they can raise an issue in Confirm just as they always would have. But now, that will also create a report on FixMyStreet which residents can view, keeping everyone up to date and ensuring that reports aren’t made about issues that the council already know about.

Canned responses

FixMyStreet Pro also allows for council administrators to create template responses — an invaluable timesaver when responding to one of the more common situations such as “issue identified and prioritised” or “issue now fixed and closed”. While Confirm also has its own template responses, FixMyStreet Pro offers more flexibility, as the same template can be reused across multiple report categories and status types. Buckinghamshire really saw the benefit of this: they were able to reduce the number of templates in use from around 450 to 46.

Mapping assets

Assets such as streetlights, grit bins and gullies can be pulled through from Confirm and overlaid on the map. This makes it significantly easier for both residents and staff to locate and report issue, speeding up the issue resolution time — we’ll be delving more deeply into this in our next blog post, with a few more technical details for those who are interested.

Image: Highways England (CC by/2.0)

We’ve upped the security on FixMyStreet

We’ve recently introduced some stronger privacy and security measures on FixMyStreet, to make things safer for everyone. They also have some nice knock-on effects that help you with moderation.

Privacy

If you’re a FixMyStreet Admin, you can now:

  • Make a user anonymous across the site, so even if they’ve made multiple reports, their name won’t show on any of them on the live web pages. Removing users’ names is a frequent request, especially from those who may have strong personal reasons not to be identified online. Users already had the ability to anonymise their reports singly or in bulk themselves, but sometimes it’s easier to do it for them, particularly if they are distressed when making the request.
  • Remove a user’s account details entirely An important point in the forthcoming GDPR regulations is that we all have the right to request the removal of our personal data from databases. In this case, the user’s reports and updates remain, but not only is the name removed from public webpages as per the point above; their email address, phone number and any other personal data are scrubbed from our own servers, too, leaving no record.
  • Hide all a user’s reports/updates from the live site. In the event that you discover a large quantity of, say, abusive reports from the same person, you can now remove them all from the online environment at a single stroke.

Security

Security for users was already very good, but with the following improvements it can now be considered excellent!

  • All passwords are now checked against a list of the 577,000 most common choices, and any that appear in this list are not allowed.
  • Passwords must now also be of a minimum length.
  • If you change your password, you have to input the previous one in order to authorise the change. Those who haven’t previously used a password (since it is possible to make a report without creating an account), will receive a confirmation email to ensure the request has come from the email address given.
  • FixMyStreet passwords are hashed with an algorithm called bcrypt, which has a built in ‘work factor’ that can be increased as computers get faster. We’ve bumped this up.
  • Admins can now log a user out of all their sessions. This could be useful for example in the case of a user who has logged in via a public computer and is concerned that others may be able to access their account; or for staff admin who share devices.

Still got any questions about privacy or security? Drop us a line and we’ll be glad to answer them.


Image: Timothy Muza (Unsplash)

Multi-selectin’ fun on FixMyStreet

As a user or a council, it’s quite possible that you’re already enjoying one of the usability improvements that FixMyStreet version 2.0 has brought — but, as it’s a fairly subtle change, perhaps without actually even noticing it.

multi-select

 

In these days of eBay and department store shopping, we’re all quite used to refining results through the use of multiple checkboxes.

But for FixMyStreet, we hadn’t given much thought to letting you filter reports by more than one dimension, until Oxfordshire County Council suggested that it would be a useful feature.

For quite some time, you’d been able to filter by category and status (“Show me all pothole reports” or “Show me all ‘unfixed’ reports”), but this new functionality is more flexible.

You can now select multiple categories and multiple statuses simultaneously (“show me all pothole and graffiti reports that are unfixed or in progress”) — and all through the power of tickboxes.

If you’re a non-technical person, that’s all you need to know: just enjoy the additional flexibility next time you visit FixMyStreet. But if you are a coder, you might like to read more about how we achieved this feature: for you, Matthew has written about it over on the FixMyStreet Platform blog.

If you’d like to know more about all the features we’ve recently introduced to FixMyStreet, why not join one of our regular Friday webinars?

Peak performance

At mySociety we believe in an open, inclusive web and such we try to build web apps that are accessible in the broadest sense. So while we do care deeply about things like WAI and the Equality Act this post isn’t about that — this is about making a site that works if you have a weak connection or an ageing device. I’m talking about performance.

Graph showing total transfer data for mobile webpages in last year.Now while it isn’t a great metric to track, the fact that the average size of a web page is now over three megabytes (and pages served for mobile devices reaching an average of 2.9mb!) demonstrates that this is an age of bloat that assumes good broadband or 4G connectivity and we don’t think that’s right.

As an example here are some numbers about the FixMyStreet site as it displays on mobile after some recent improvements.

To load a working and styled front page on your phone takes around 9KB of HTML/inlined CSS/inlined images (that isn’t a typo – nine kilobytes). How do we pull that off? Well, the site logo and menu are both inlined so we don’t have to wait for them to load, as is the CSS needed to show the top part of the front page. 5KB of JavaScript is loaded (which amongst other things enables the geolocation) and in the background an additional massive 14 kilobytes of CSS (the main mobile stylesheet) and the remaining 20 kilobytes of images (the example report photos and footer links) are being pulled in. The page also uses prefetch to start fetching the remaining JavaScript while the user is entering a postcode or address to actually get started on FMS.

On a desktop there’s a little bit more to add to the mix (more like 66KB of images, 19KB of CSS, plus a webfont taking 77KB) but it’s still lightning quick.
The team haven’t reinvented the wheel to achieve this – they’ve just been ruthless and absolutely focused on only using the minimum amount of code to meet the user need. When the FixMyStreet site is deployed, the JavaScript and CSS is automatically minimised, and at that point we run penthouse> to work out the critical CSS to be inlined on the front page. And whilst our main JavaScript does use jQuery, we dropped it from the front page to save yet more up-front time (jQuery is far larger alone than our current front page).

If you are interested in more details of how this was achieved, here’s a post Matthew prepared earlier on many of the same techniques, which he used on his own project traintimes.org.uk.

There are of course still improvements to be made – I imagine many front page viewers of FixMyStreet never need or want to scroll down as far as the images in the footer, so ideally we wouldn’t load them unless they do. Due to Windows Lumia users, which we support for a specific client use case, we’re using Appcache for offline support, but adding some form of more modern service worker would also be nice. And most of this work is for the front page (though it helped other pages too); our main JavaScript could be split up more than it is. It’s a continual process, but here is a good place to pause.

Photo by Braden Collum on Unsplash

FixMyStreet reaches Borsetshire

What would Eddie Grundy do if he came across a pothole? And how would Linda Snell deal with flytipping on the site of the Ambridge village fete?

Fortunately, these fictional characters now enjoy the same access to FixMyStreet as the rest of us, thanks to the new demo site we’ve built.

The thinking behind it is not, of course, to gather reports from an entirely fictional world. We’re not that mad.

Rather, we needed a sandbox interface where we could show councils exactly how FixMyStreet works, and allow them to play about with both the customer end and the admin side, all without causing any major repercussions to the running of the standard site. Enter FixMyStreet Borsetshire.

Prospective buyers of the system from local councils can experience the various levels of administration that the back-end allows. Just log in with the credentials seen on this page and see exactly how reports can be shortlisted, actioned, or moderated.

So, we’re expecting reports of pigs on the loose, flooded culverts and perhaps even a flying flapjack. But if you’re hoping to find out the precise location of Ambridge, unfortunately you’ll be disappointed: the map is actually centred around Chipping Sodbury, far from the village’s supposed Midlands locale.

Keen to know more? Join one of our regular Friday webinars for the full tour.

Image: Martin Pettitt (CC by/2.0)

How councils get extra benefits, at no extra cost

We’re always making improvements to FixMyStreet. And the great thing is that, as a council that’s installed FixMyStreet Pro as your fault-reporting system, you get all those improvements for free.

With most SAAS providers, there’s no real incentive to keep refining a product once it’s sold — but FixMyStreet is a bit different. The same software that underpins FixMyStreet Pro also drives our nationwide site for the UK, FixMyStreet.com, and a number of FixMyStreet sites run by other groups across the world. When we make an improvement to the codebase, that’s rolled out so that everyone can benefit from it.

The latest updates can always be seen on the FixMyStreet Platform blog, where, as you’ll see, some recent additions include the potential to update reports by text rather than email, and some performance updates as well as bug fixes.

Because FixMyStreet is in a continual state of development, we are always happy to hear from councils about the features they’d like to see. Just let your account manager know, or, if you’re familiar with the site GitHub, you can raise a ticket directly on the public FixMyStreet repo.

And when we make announcements on the blog, if you see anything you’d like to carry across to your own installation, drop us a line and we can discuss how best to make it happen.


Image: Štefan Štefančík (Unsplash)

Something in the middle: how Bristol connects

This year, Bristol Council did something unusual and admirable. As far as we’re aware, they’re the first UK council to have taken such a step.

Working with mySociety on custom Open311 ‘middleware’ while adopting FixMyStreet as their fault-reporting system, they now enjoy full flexibility, no matter what the future holds.

Thanks to this open approach, Bristol will extract more value from their existing systems and lower operating costs. With integrated, open solutions, and the raised quality of report formatting that Open311 brings, everyone will benefit.

Improving flexibility

Councils are increasingly understanding the value of flexibility when it comes to service providers.

Contracts that lock them into a single provider for many years mean that, often, there’s no opportunity to benefit when technology advances, and disproportionate costs can be charged for implementing the slightest changes.

This desire for flexibility was a strong factor in Bristol City Council’s decision to adopt FixMyStreet for Councils — and that opened the door for a conversation about Open311.

We’ve always advocated integration via Open311, to the extent that we offer free hook-up with FixMyStreet to any councils who support it.

Because Open311 is an open standard, it supports the entire landscape of providers like FixMyStreet. Right now, Bristol can accept street fault reports not just from us, but from a full range of services — in other words, any site or app that cares to connect with them can do so. No-one knows what the future will hold: if a game-changing system emerges in the future, it makes sense that you’d be able to accept its reports.

All well and good: but when Bristol City Council implemented FixMyStreet as their fault-reporting system, the concept was taken a little bit further. With our collaboration, Bristol created their own Open311 ‘middleware’, sitting between the two systems and talking to both.

Via this method, their existing CMS, Confirm, can hook up to reports coming through from FixMyStreet. That all works smoothly — but, just as importantly, if Bristol ever decide to replace their CRM provider, they’ll be able to do so with no knock-on effect to FixMyStreet reports. And if they ever decide to replace FixMyStreet with a different provider, or indeed to accept reports from a range of providers, they can do that too.

Bristol found us via the GCloud procurement system, and are the first metropolitan unitary authority to install FixMyStreet.

Future plans

Bristol launched its FixMyStreet service to the public in the summer of 2016.

This autumn, they added asset-based reporting, meaning that known council properties such as streetlights, grit bins and gullies are all marked on FixMyStreet’s maps. Residents can pinpoint and report the location of faults with these assets far more accurately as a result.

There’ll be a phased rollout across departments, starting with Highways and moving across departments as Bristol extend their own middleware. We’ll be watching with great interest.

Interested to learn more? Join one of our regular Friday webinars.



Image: Adam Heath (CC by-sa/2.0)

FixMyStreet version 2.0

The FixMyStreet codebase is used all over the world by people running versions of the site for their own country or jurisdiction. This week, we’re proud to announce the release of FixMyStreet version 2.0.

This version contains a wide array of new features that benefit FixMyStreet sites’ users, administrators, and the officials who receive reports. They include elements that the UK FixMyStreet was the first to trial, such as nicer-looking HTML emails for users and authorities, the ability to filter reports by multiple states and categories, a new admin user system with graduated permissions, and various bugfixes and development improvements.

Over the next few weeks, we’ll be publishing a series of blog posts over on fixmystreet.org/blog/, examining the changes in detail. If you run a FixMyStreet site, or you’re just interested in coding and technical issues, you may find them of interest. Meanwhile, here’s the broad overview.

Image by Romana Klee

New front-end feature

HTML email

There is now the option for all emails sent by FixMyStreet to be HTML formatted where previously they were plain text only. This includes confirmation and questionnaire emails to the user, and report emails to the public body. These emails include any image added to the report, plus a small static map of the problem’s location.

State/category filtering and sorting of list pages

When viewing a list of reports, you can now filter and sort them in pretty much any way you choose, including sorting by most- or least-recently updated, newest or oldest, or most commented. You can also select multiple categories or states (e.g. “fixed”).

 Pretty area highlighting on body pages

The highlighting of areas on a body page has been inverted, so that the unimportant parts of the map are shaded and you can interact more easily with reports on the page.

Users can now update their own email address

This was a frequent request from users and we’re glad to report that they can now do it themselves on their account page.

Performance improvements

When looking at reports from a list page, the other report pins stay visible so that it is easier to switch between them. The report itself is being pulled in behind the scenes, meaning the whole page does not need to reload. The map no longer extends underneath the sidebar and header, which makes things easier, and a scroll wheel can now zoom the map in and out.

Making privacy options clearer

The reporting form has been separated into public and private sections, to make it clearer which parts of what you provide will be made visible on the site.

Showing the relevant recipient

If you live in an area where there’s more than one body, the category you pick normally dictates which body we send your report to. Now, when you select the category we update the name of the body given at the top of the report page, if we know that the report will be sent there.

 

New admin user system

Admin users can now use the same log-in right across the site – whether they’re making a report like a standard user, or logging in to make edits and moderate the site.

In the past, the distinction between admin and other users was black and white. As an admin user, you had access to every part of the site, but users can now be given individual permissions for various layers of access. These include:

  • Proxy users This layer grants the ability to create a report or update on behalf of a body, or as another user. We envisage this being useful in a body’s contact centre, where they receive a report over a phone and enter it into FixMyStreet as that user;
  • Report editors Giving the power to edit a report’s category, state, or location. If the admin user changes the category, and that change means that a different body is now responsible for the report, it will be re-sent;
  • List makers, who can compile their own shortlist of reports they wish to go and inspect. This may be useful for a contractor or team who wishes to compile the day’s tasks;
  • Quick responders These users have access to response templates, allowing them to edit and publish templated updates;
  • Prioritisers These users may set different priorities on reports;
  • Trusted users A simple reputation system, which e.g. potentially lets reports from trusted users be actioned more quickly.

The admin report edit form has also been greatly improved, including a map to update a report’s location (and re-sending the report if the body changes), and much tidier layout.

Bugfixes and development changes

Bugfixes include updating the top-level domain (TLD) list for email validation, hiding authorities which don’t exist any more on the all reports page, and fixing the previously-broken photo preview display after form submission. We have dropped support for Internet Explorer 6.

If you’re a re-user of the codebase, there are a number of changes that will hopefully help you out. See the extended version of this blog post on fixmystreet.org for more details.

If you have any questions, please do get in touch.

FixMyStreet Pro blog

FixMyStreet Pro is the street & environment reporting service that integrates with any council system.

Join our newsletter

Be the first to know about FixMyStreet-related launches and events, surprising stats, and digital breakthroughs.

Join one of our regular webinars

Show webinar schedule

Schedule your one-to-one demo

Request a callback

See FixMyStreet Pro for yourself

Try our live demo