The A-Z of Laravel & A Maze of Development Shortcuts

Engineering is a discipline which can be a harsh teacher; repetition can be porous or opaque in how it educates you to optimize, and not simply through immiseration. We have few exams, but many unspoken qualifications. At some point, a lot of technical people either resign themselves to doing it the way it is always done, or they look for ways through to more interesting pastures or faster highways.

The mark of an effective engineer is graceful simplicity. It takes time to make something simple, even if the problem it solves is complex. Anything is a mess when it is attempted for the first time.

Laravel is extremely effective in several areas: a) fast ecosystem-led prototyping, b) lowering barriers to entry, and c) flexible deployment. Where it lacks is 1) scalability, 2) support for advanced methodology, and 3) tolerance of bad practice.

What that means in reality is it helps materialize new ideas quickly, but suffocates them as they mature. There are lessons you learn along the way.

A - Asynchronous Always First

Something might need to be a queued job later when load is put on the application. Design it as a job which initially uses the sync driver. It's extremely difficult to change a blocking operation into a non-blocking one, but there is no difference in creating a sync job. Identify areas of code which may need to be queued. Requests themselves can be dealt with asynchronously using Swoole.

B - Bus The Spaghetti Away

Writing code to connect more than five APIs turns into chaos extremely quickly. If you are routing data from place to place in a way which isn't maintainable, it's time for a single point of contact: the Enterprise Service Bus (ESB). A bus can be thought of as a sound desk, with inputs, channels, and outputs. Identify the different APIs you are using and subordinate the messaging elsewhere.

C - Content Goes On CDNs

The web server already has a CPU-killing job: requesting dynamic HTML from an interpreter which includes dozens of trips to a database. There is no reason to be pulling static files from a public web root directory when they can be served from a specialised endpoint. CSS, JS, fonts, file downloads, and images, all need to live somewhere else. Uploads to one machine can't be shared with others.

D - Data Needs To Be Pruned

Your development machine never needs or generates more than a few hundred records. After six months in production, your clever logging is going to be creating gigabytes of useless data nobody cares about; whether it is in /var/log or the activity table, remember to add a scheduled task to remove or archive records more than ninety days old, as part of your Retention Strategy.

E - Eloquent is Not A Cacheable Repository

Laravel was inspired by Ruby-on-Rails, and its database engine is an implementation of ActiveRecord: a single row in a relational database is represented as a software object. Eloquent mangles the Repository Pattern in a mutated, horrible way with static methods like all() and first(), moreover fails to include any form of effective caching. Investigate true repository implementations which abstract it.

F - Flat Files Should Serve Reference Data

The rule is if you're not updating something more regularly than every six months, it's faster to serve it from a static configuration file than a database. There is no reason to have twenty tables providing countries, currencies, languages, locales, and so on. If it can be placed in a static file, remove the database round-trip.

G - Give Grafana The Graphs

Every developer hates the complexity of graphs, even when using chart packages to make it simpler. Graphing is a specialised practice which is highly labour-intensive and subject to enormous personalisation on the whim of a client. Grafana server allows your client to connect directly to the application's raw data and create their own dashboards instead of resorting to Tableau. Set them up with a docker instance.

H - HTTP Client APIs Need Sandboxes & Automated Testing

It's not enough to write resource controllers with live data. Your clientele aren't going to implement your Swagger API in a day and will need to experiment heavily while integrating, just as you do. You might be able to stage, but what you will actually end up needing is a dummy environment for each API client to read and write data from which doesn't destroy everything. And you'll need a way to automate unit tests of your API endpoints which is separated from Laravel.

I - Install Commands Help Newbies

Clone the repo; download the packages; migrate the database; seed the dev data. It doesn't have to be like this. When you are bussing in dozens of developers, the most helpful command you can have is app:install. There is no need to supply a giant list of commands in a readme document which take four hours to debug when all of the steps can be created inside one.

J - JSON Can Go Very Wrong

It may be the universal interchange format, but what if you are serializing emojis, Cyrillic, or, special characters, or unstructured data? Laravel doesn't serialize properly because it forgets the important options json_encode provides, such as JSON_HEX_QUOT, JSON_HEX_TAG, JSON_HEX_AMP, JSON_HEX_APOS.

K - Kill All Your Clever Ideas

Python's philosophy makes an important distinction between "complex" code and "clever" code. Complexity is not a vice, nor is it to be feared. "Clever" code is a result of uncontrolled ego, and wanting to impress others; the unrestrained product of hubris. Two lines is better than two hundred. Play some Perl Golf.

L - Logic Doesn't Belong In Controllers

In the Model-View-Controller pattern, on which Laravel is based, a Controller passes data from a Model into a View. It's Single Responsibility is to mediate the passage between the information persisting in data storage (usually a database) to a visualisation on a screen. It is not there as a place to house procedural code. Your logic should be created as Actions, Modules, Processes, and Services .

M - Multilingual From the Beginning

Most programming languages were designed in English, but the Internet is global. Laws in Quebec, for example, demand all signage, technology, or documentation, are presented in French first, then English. A third of the world speaks Spanish. Most of the Middle East uses a character set which has more letters than twenty six, and goes in the opposite direction. If you ever have to reverse engineer four hundred Blade files to use language variables, you will wish you used them up-front.

N - N+1 Problems Are Unprofessional

Relational databases allow joins and views; NoSQL databases allow storing entire embedded objects. In the days of SQL statements, it was horrifically common to run thousands of loops for related or nested data. Laravel includes a simple technique called Eager Loading which performs a simple WhereIn() query to avoid those thousand database trips. But it can fail miserably if your related data is 100,000 rows long, so apply limits.

O - Opcache Speeds You Up

PHP is an interpreted language, like Perl or Ruby. It exists as static text files which need to be run through a binary program every time a request is made, in comparison to compiled languages like the C family, Go, or Java. PHP 8 now includes a Just-In-Time (JIT) compiler which can be switched on so those text files are compiled up-front. The price is caching - your exceptions are remembered even when you deploy new code and forget to reload it.

P - Proxies Change Your Environment

When you reach a few thousand requests a second, you are going to need to containerise your application and use a load balancer (e.g. HAProxy or Traefik) to route requests to available nodes. Suddenly, different machines have different sessions; cookies are set for different machines; IP addresses are different each time and not the end user. Plan ahead for centralised data storage, network latency, and validating who is sending the request to each container.

Q - Queues & Schedules Need Observability

By their nature, queues run in the background and the jobs they run fail more often than they succeed. The price of their usefulness is the inability o to visualise them. Laravel Horizon only works with Redis, and there is no real way to watch the scheduler run. Delegating logic to backend services requires serious thought to DevOps; or how other people who aren't you intend to observe your code running in real-time outside of the dashboard.

R - Remote Databases Help Team Development

Seeding data isn't fun, ever. Whether you are writing it, installing it, or God forbid, attempting to share a version-controlled set of it. The simple solution is to create a remote, centralised database on a VPS or Cloud platform all devs share over TLS which is your "experimental" environment. The price is if a junior does something stupid, everyone suffers for it, so, create several, and place warnings.

S - Subdomains Create Conceptual Separation

Typically, you are going to have at least four domains to your application: the front ("marketing") brochure, the dashboard application, your admin control centre, and your REST API. And about six months in, you are going to realise it's extremely difficult to put these under the same roof, despite them sharing the same database. Laravel allows you create separate subdomains in its routing which can allow you to separate your application folders logically, in preparation for when you need four different Laravel applications later.

T - Timezones Make UTC Look Good

The US has at least three different timezones (EST, CST, PST), even without counting its dependent territories. Databases always store timestamps in UTC (GMT), and present the converted date and time to users visually after it has been retrieved. If you think that's a flawed strategy, try doing it the other way. User accounts need to store timezones from the beginning, so they can apply that transformation; datetime pickers in JS need to convert specified times back into UTC when performing inserts.

U - UR HaXX0r Code Is Unreadable

camelCase iS aNnoying tO rEad iSnt It. its_much_easier_to_read_snake_case. Do you really have to getData on a getConnection function which uses getApp and have a setSomething for every attribute? Someone is going to have to maintain your code afterwards, and they will read from left to right, generally speaking. No, the simplicity of your code should not require comments, but in the cases where it is complex, simple politeness demands notes to help the next guy.

V - Views And Stored Procedures Replace CRUD

One thing Laravel is appalling at is the basic concept of database views and stored procedures. All major databases - relational or NoSQL - feature the ability to create views of aggregations, and standardise methods of inserting data. Down the road, you are going to need to set back your logic into the data layer to avoid destroying the server's CPU with calculations. Database views can be used as migratable, read-only "tables" in Eloquent models, and folder-friendly database Schemas can help organise the mess. Postgres and Oracle can materialise the data to disk when calculations become too intensive.

W - World-Changing Starts With Debugging Servers

Make change; refresh page; see exception; alter code; dump; loop. NPM might be able to "watch" your files for changes and Live Reload might apply them in the browser, but the world of blocking, synchronous request lifecycle doesn't have to be painful. Using a debugging server like XDebug in combination with Seq introduces a new way to look at everything. Set your logging up as an interactive console environment.

X- Xtra Validation Is Needed for Telephone Numbers

Everyone formats telephone numbers differently. 555-123-000 is the same as 555.123.000 which is the same as +0015551231000. What about extensions? International dialing codes? What if they change, or are an invalid? Just as databases store time as UTC, you can store international phone numbers with extensions as E164 and validate them using Google LibPhone library.

Y - Yoga Is No Substitute For Camaraderie

Ultimately, development is about relationships and trust. When you trust someone, you don't need to manage them, even if they might get something wrong. If your relationship is based on honest communication, your problem solving is just a brain-storming session. People don't know things; they get things wrong; they don't go fast enough; they forget to remember what they should have done; they want to be proud of what they've done. All of that flows from being able from fun, humour, and the freedom to get it wrong. Camaraderie helps you suffer well.

Z- Zealotry About Security Is Essential

It's not practical in development to apply security first, as much as people might want you to. We have full access to our own machines, so it isn't an issue. However, four specific places need enormous care: a) .env files need to be stored in a transferable vault, b) staging servers should be on a VPN, c) sensitive database fields should be encrypted in case the database server is broken into, and d) code should be scanned for problems with a tool like SonarQube.

We have 10,000 graphs to create.

Don't bother. Hook up your database to Grafana: (https://grafana.com) and let your client create their own dashboards.

We have massively complex code producing huge data reports.

It's time to put the logic back into the database and use Views. Create your joins etc in the database, and materialize your monolith chaos as a Materialized View (https://en.wikipedia.org/wiki/Materialized_view) with a unique index so it can be refreshed every few minutes. Better still, use NoSQL aggregation pipelines (https://docs.mongodb.com/manual/core/aggregation-pipeline/).

We want to use Docker.

Install Docker and Docker-Compose. Then go for Laradock: https://laradock.io/.

We need a massive, YUGE, enormous database.

You don't need a database, you need a cluster. And that means Apache Cassandra (https://cassandra.apache.org/) when they get round to updating the PHP extension: https://datastax.github.io/php-driver/1.2/building/ .

We want to build our own PHP extension.

Not the best idea but if you're insistent, the quickest way is using Zephir: https://zephir-lang.com/en .

We need to talk to USB, Bluetooth, and VR headsets from the browser.

Take a look at WebUSB (https://wicg.github.io/webusb/), Web Bluetooth (https://webbluetoothcg.github.io/web-bluetooth/), WebXR (https://www.w3.org/TR/webxr/), Vibration (https://www.w3.org/TR/vibration/), WebNFC (https://www.w3.org/TR/nfc/), Web Serial (https://wicg.github.io/serial/).

We don't want to keep SSHing in to debug errors.

Then you're going to need to install Graylog (https://www.graylog.org/, MongoDB + Elasticsearch) or Seq + GELF input (https://datalust.co/seq) in a Docker container.

We need to cache static output of HTML/JSON.

If it's read-only and unlikely to change, you'll need Varnish (https://varnish-cache.org/). It doesn't do SSL, so you'll need something like Pound (https://www.apsis.ch/pound.html) or HAProxy (https://www.haproxy.org/) in front of it.

If it's time to leave the password manager, look at Envault (https://envault.dev/) or Zookeeper (https://zookeeper.apache.org/).

We need to secure a program which doesn't have SSL.

You need sTunnel (https://www.stunnel.org/) or Nginx as a reverse proxy with basic auth (https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/).

We need to virus-scan uploads.

Use Clam-AV (https://www.clamav.net/) in your request validator: https://github.com/sunspikes/clamav-validator

We need to do facial detection/recognition in Laravel.

The quickest way is AWS Rekognition (https://aws.amazon.com/rekognition/) but if you are rolling your own, start with OpenCV (https://opencv.org/), FacePlusPlus (https://www.faceplusplus.com/), and Open Biometrics (http://openbiometrics.org/).

We want to do Machine Learning in Laravel.

Bad idea. You'll want to start with PHP-ML (https://php-ml.readthedocs.io/en/latest/) and Rubix ML (https://rubixml.com/). For NLP, there is NLP Tools (http://php-nlp-tools.com/)

We want to build a command line program like Composer with Laravel.

Use Laravel Zero (https://laravel-zero.com/) which creates executable Phar archives using the Box Project (https://github.com/box-project/box).

We need our staging servers to be on a VPN.

Use Wireguard (https://www.tailscale.com/) or OpenVPN (https://openvpn.net/). If you need a privacy proxy, use Streisand (https://github.com/StreisandEffect/streisand). For OpenVPN, use a setup installer, copy your .ovpn file to /etc/openvpn/something.conf and run systemctl start openvpn@something.

We need to load-balance Docker containers.

Use Traefik: https://traefik.io/traefik/.

We need to check text for profanity and writing quality.

Don't attempt to build a profanity filter. Use WebPurify (https://www.webpurify.com/documentation/) and the ProWritingAid API (https://prowritingaid.com/en/App/API).

We need to customise Nginx.

Nginx is small and fast because it's designed that way. If you want to do more advanced programming in its native language (Lua), you will need to look at Open Resty (https://openresty.org/en/).

We need to build a "find-my-nearest" app.

Then you're creating a geometric proximity query service which uses the Haversine Equation (https://en.wikipedia.org/wiki/Haversine_formula). You'll need a package (https://github.com/eleven-lab/laravel-geo) which works with a geospatial index on a PostGIS (https://postgis.net/) or NoSQL database plugin.

We need our own Git server.

Use Gitlab (https://about.gitlab.com/).

We need to do SSO and SAML.

Good luck. Your best bet is to reserve a server to run Gluu (https://www.gluu.org/).

We want to do advanced image processing and url-based manipulation.

For interpreting things like PSD, EPD etc as well as complex effects you can't do in HTML5, there's ImageMagick (https://imagemagick.org/). For Cloudinary-style hosting, you can do worse than Thumbor (http://thumbor.org/).

We want to build a desktop application in PHP.

Why? Use PHP-Qt (https://www.openhub.net/p/php-qt), or PHP Desktop (https://github.com/cztomczak/phpdesktop). Most modern approaches use ElectronJS (https://www.electronjs.org/).

We need to do chat and instant messaging.

Don't write it yourself. If you can't use an external API, look at programs with Websocket integration like eJabberD (https://www.ejabberd.im/) or Openfire (https://igniterealtime.org/projects/openfire/).

We want to use Websockets.

Laravel comes with Pusher out of the box, but you essentially have two alternatives: Crossbar (https://crossbar.io/, Python), and Centrifugo (https://centrifugal.github.io/centrifugo/, GoLang). Both have JS libraries but the latter comes with JWT authentication.

We don't want to send test emails to real email accounts.

Use MailTrap (https://mailtrap.io/), MailHog (https://github.com/mailhog/MailHog), MailDev (https://maildev.github.io/maildev/), or set up your own Dovecot (https://www.dovecot.org/) server.

We need a free code quality analyzer.

Use the open-source tool SonarQube: https://www.sonarqube.org/

We need to do server-side document scanning.

You'll need Tesseract (https://github.com/tesseract-ocr/tesseract) for OCR, LibreOffice's Uniconv (https://github.com/d-mozulyov/UniConv) for converting documents, and something like a PDF library (https://www.xpdfreader.com/pdftotext-man.html).

We need our own actual, real search engine instead of a database.

Good. Databases are not search engines. The gold standard is Elasticsearch (https://github.com/babenkoivan/scout-elasticsearch-driver), but there is also Manticore (https://manticoresearch.com/) if you need a shortcut. It's not as difficult as you think.

We want our own version of Amazon S3.

Use Minio (https://min.io/).

We need a better version of Supervisor.

It's been ported to Go: https://github.com/ochinchina/supervisord . Also think about using forked SystemD unit files: https://unix.stackexchange.com/questions/516749/how-best-to-start-my-systemd-service-to-run-multiple-apps

We need to set up a lot of version-controlled database servers.

Then you'll need a way of versioning them like LMM (https://github.com/Lullabot/lmm). You can use Liquibase (https://www.liquibase.org/) to manage the changes, and a read/write-configured proxy like MySQL-Proxy (https://github.com/mysql/mysql-proxy) for local devs.

We need to take screenshots of web pages.

This is done with a "headless" browser taking a screen capture. You can use Puppeteer (https://github.com/puppeteer/puppeteer) or WKHTMLtoPDF (https://wkhtmltopdf.org/)

We're thinking about using a graph database.

Use Neo4J (https://neo4j.com/) with the Eloquent extension (https://github.com/Vinelab/NeoEloquent).

We need to build an IoT application.

Then you will need a MQTT library (https://mosquitto-php.readthedocs.io/en/latest/, https://github.com/salmanzafar949/MQTT-Laravel) and backend server like Mosquitto (https://mosquitto.org/) or HiveMQ (https://www.hivemq.com/). You'll probably need Influx and Telegraf (https://www.influxdata.com/) too.

We want to do our own document signing.

You'll need to start with SignServer (https://www.signserver.org/).

We need to do automated video encoding.

You'll need a package to manipulate FFMPEG: https://github.com/protonemedia/laravel-ffmpeg which puts items into queues.

We need to apply complex audio effects to sound files.

You need SoundeXchange or "SoX" (http://sox.sourceforge.net/), which you can use with a command line package like Symfony Console.

We need to stream our own audio and video.

For audio, use Rocket Broadcaster (https://www.rocketbroadcaster.com/streaming-audio-server/). For video, use Wowza Media Server (https://www.wowza.com/). Have a look at Gearman (http://gearman.org/) for managing processes.

We want to build our own mapping service.

Use OpenStreetMap (OSM, https://wiki.openstreetmap.org/wiki/Main_Page) and its tools. If you want your own Gmaps to do your own cartography, look at MapServer (https://mapserver.org/).

We need to talk to desktop programs.

If you're not using COM extensions on Windows (https://www.php.net/manual/en/book.com.php), many of them come in "headless" versions, like Adobe InDesign Server (https://www.adobe.com/products/indesignserver.html)

We can't get access to a server behind a NAT router.

Use Ngrok (https://ngrok.com/) for simple requests (e.g. Web hooks) or go the whole way by getting the machine onto OpenVPN: https://www.ubuntupit.com/how-to-install-openvpn-in-ubuntu-linux-a-tutorial-for-newbie/ .