Today running a Lightning Network node presents operators with a series of tradeoffs and operational requirements. Nobody said building the future of financial infrastructure would be easy, after all. Two tasks in particular seem to dominate: liquidity management and channel management. The objective of liquidity management is to ensure that a node has enough capacity to send and receive funds. Channel management concerns the opening and closing of channels, to ensure that a node is sufficiently connected to the network, and its capital is efficiently allocated.

We have covered the topic of liquidity management extensively in our loop out and loop in blog posts. In this post, we focus on channel management, covering the concepts of good and bad channels, as well as some criteria for making channel close decisions. Today we’re excited to announce the Faraday tool to provide node operators with recommendations for closing channels that are not performing, and to expose more information about your node's routing behaviour. Throughout this post, we include code blocks containing examples of how this tool can assist in managing your node’s channels.

Uncertain Peer Behaviour

In A Routing Node's Guide to the Galaxy Part 2 we discussed some of the network-level indicators that can be used to identify a good peer. These indicators — channel age, node capital and number of channels — provide a good approximation of which peers should be better to connect to, and which should be worse. Nodes in the network don’t share information about peer behaviour, so the amount of reputation data that can be gathered is limited to the publicly available information exposed by the protocol.

Although you may consult node scoring algorithms such as Lightning Labs’s Bos Scores before opening your channel, you have no guarantees of how they will behave once you have committed your funds. The peer's behaviour only truly becomes apparent after a channel has been opened. Many node operators find themselves in the situation where they’ve opened a channel, only to have their peer disappoint. The channel may route little-to-no traffic, the peer might be unreliably online, or the peer may simply go offline, never to be seen again.

Low Volume

Public channels in the Lightning Network commit capital for routing in exchange for fees. While BTC is held in a channel it cannot be used for other purposes; it can be freed up by closing the channel out on chain at any point, this requires on chain fees, waiting for the closing transaction to confirm and a possible delay to accessing your BTC. If a channel is processing little-to-no volume, it will be earning little-to-no fees. Low volume is an indicator that your channel is not being utilised by the network.

Payments routed over the Lightning Network are private. Unlike the base on-chain layer, where every transaction is broadcast to every node, Lightning nodes have no idea what volumes the network is currently processing. Most mobile nodes connecting to the network do so via private channels, which Bitmex Research recently estimated account for 27.8% of the channels in the network, albeit holding only 12.7% of the BTC on the network.

Choosing the level of expected volume is challenging given this limited information set. We have added outlier-based close recommendations in Faraday which can be used to identify the channels that have low volume relative to the volumes your other channels are processing. In order to account for channels that have been open for different periods of time, the values provided are scaled by the number of confirmations (channel age) that each channel's funding transaction has. In this example output, one channel is routing significantly lower volumes than the other channels, which identifies as a candidate for close.

./frcli outliers --volume { "recommendations": [ { "chan_point": "0bc094848a3400959232397b0e5252c1ec8711333517ab858fe83b81087bada4:1", "value": 32.4929, "recommend_close": true }, { "chan_point": "179a0ecf39476966e9d22a659555e1f89f06f25f71cbc6ee980bfd8f1039be5f:0", "value": 205.5729, "recommend_close": false }, { "chan_point": "1a2f48953a3c8402d7afa9f16de84a2dc7491876dc0e0b7ed05fe30b565dd656:1", "value": 300.869565, "recommend_close": false }, ] }

Effective Liquidity Management

Lightning channels do not exist with a peer in isolation. If you wish to forward payments out over a channel, it requires that your peer have outgoing capacity that the payment can be sent out on. Likewise, for you to send payments onwards in the network along your channel, those payments need to have arrived at your peer via a channel that has incoming liquidity. If your peer does not manage their other channels, it will affect the ability of your channel with them to route payments. A peer that maintains a balanced set of channels is more valuable than one who does not actively manage their channels.

Channel balances on either side of a channel are kept private in Lightning to preserve privacy of the participants, so we cannot determine whether our peer is maintaining balanced channels other than our own. Although this information is not available, we do have a better idea of whether a node's channels are depleted when we have a channel open with it, because we participate in forwards with that node. If you see a sudden drop off of incoming payments from your peer, this may be because they have depleted their incoming liquidity and payments are failing before they reach you. The same can be surmised of a sudden spike in forwarding errors from your peer. The next release of lnd will contain a htlc event subscription which can be used to more closely monitor peer forwarding patterns.

Peer Uptime

A channel has bad uptime if we cannot communicate with the remote peer that we have the channel with. This may be because the peer's networking is misconfigured, or simply because their node is not running. The Lightning Network processes payments through the update of a commitment transaction between peers. If your peer is offline, this commitment transaction cannot be updated, and therefore the channel can’t process any payments. Channels with offline peers are of no use to the network because they can’t facilitate further forwards. If a channel has been offline for more than 20 minutes, lnd nodes inform the network that the channel is disabled. This policy only applies to public channels, which are advertised to the network with the intention of being used by other nodes to route payments. Private channels, which are primarily used by mobile nodes to connect to the network, are not used for routing payments other than ones to/from the private channel peer, so lower uptime is acceptable.

Peer uptime is a proxy for how reliably you’ll be able to use a channel with that peer. If they’re only online 30% of the time, you'll only be able to use that channel for roughly 30% of the time. Your uptime requirement will depend on how critically you require reliable lightning network payments; if you’re using your node for personal use only, lower uptime is more acceptable than operators running production-level nodes. To allow for easy identification of peers that do not meet this required threshold, we have added a threshold endpoint to Faraday which allows you to filter for peers that do not meet your requirement.

./frcli threshold --uptime=0.6 { "recommendations": [ { "chan_point": "7649fa4a7672d7cf7ec5a94dd6092a36e2c16d048fd164170a77fd7a9d118eb2:0", "value": 0.502, "recommend_close": true }, { "chan_point": "87346994ed40d6487f9b1a3ba57406b4e751d96416e80b823afd469a54d76682:1", "value": 0.87, "recommend_close": false }, { "chan_point": "0bc094848a3400959232397b0e5252c1ec8711333517ab858fe83b81087bada4:1", "value": 0.96, "recommend_close": false }, ] }

At present, uptime tracking is not persisted in lnd, meaning that when you restart your lnd node, its record of peer uptime starts over. When making uptime-based close decisions, you want to be sure that uptime data has been gathered for a sufficiently long period of time; 100% uptime over one minute tells you little about your peer’s quality. To control for this, we have included a minimum_moniotred cli flag which allows you to specify the minimum amount of time a channel should have been observed before it is included in these recommendations. By default, this value is set to four weeks.

Changes to Fee Policies

Each peer in a channel is entitled to set their own fee policy. Your fee policy will determine the fee that is paid to forward payments through your node and onwards to your peer, and your peer's fee policy will determine the fee that payment pays to reach you over the channel that you share. Pathfinding algorithms for lightning payments aim to minimise some combination of fees and timelocks duration along a route. If your peer increases their fee schedule, it may affect the number of forwards your node processes, because pathfinding will route around channels with overly expensive fees. The Describe Graph rpc endpoint can be used to examine your peer's fees relative to the rest of the network and GetChanInfo can be used to track the fee schedule for an individual channel.

If You Love Your Node, Let Bad Channels Go

Once you’ve opened a channel, it’s tempting to leave it open in the hope that your peer will eventually come right. Closing of Lightning channels is often cast in a negative light. After all, the Lightning Network achieves scale by processing many off-chain payments at the on chain cost of two (worst case three) base layer transactions. The more on-chain transactions we see, the lesser the degree of scale the Lightning Network can achieve. This statement is true of productive channels, which could have been used for further off-chain payments if they'd remained open, but does not hold for non-productive channels.

If you have non-productive channels, as identified in the section above, it is beneficial for you as a node operator, and the network as a whole, to close these channels. Reallocation of capital can increase the fees your node earns from forwarding payments, and potentially decrease the number of hops a payment requires if the new channel sits on an underfunded path. Base layer fees are also low right now, but will not remain so forever. Now is an opportune time to close out unused channels at a low on-chain cost.

Channel Churn vs. Channel Cleanup

The context in which channels are closed matters. The closing of a bad channel has little negative effect on the network, because it sees low usage by definition. However, if a channel that was forwarding payments successfully on the network is closed, this can be considered a bad channel close. Channel closures of this nature often occur due to operational concerns, such as the need to acquire outgoing capacity on a channel that has been spent down. The closing of a channel only to reopen another channel to the same peer is called 'Channel Churn'.

While this strategy achieves the end goal, it comes at the expense of an additional two on-chain transactions. The same outcome can be achieved off-chain through circular rebalancing, or with batched on-chain transactions using a submarine swap service, such as Loop In which allows your channel to remain open.

Know Thy Node

If you’re not ready to take the axe to your current set of channels just yet, we’ve provided some purely informational endpoints in Governator which will provide further insight into how your node is operating. All of the metrics used for close recommendations are exposed on an insights endpoint:

./frcli insights 179a0ecf39476966e9d22a659555e1f89f06f25f71cbc6ee980bfd8f1039be5f:0 { "chan_point": "179a0ecf39476966e9d22a659555e1f89f06f25f71cbc6ee980bfd8f1039be5f:0", "monitored_seconds": 259032, "uptime_seconds": 127008, "volume_outgoing_msat": 1000, "fees_earned_msat": 500, "confirmations": 18, "uptime_ratio": 0.49031779857, "revenue_per_conf_msat": 27.77777777777778, "volume_per_conf_msat": 55.55555555555556, "incoming_vol_per_conf_msat": 0, "outgoing_vol_per_conf_msat": 55.55555555555556 }

Revenue and Volume Report

Payments in the Lightning Network tend to flow through the graph with direction; some nodes will mostly forward payments onwards to you, acting as the incoming peer in the forward, others will often be the node that your payments are forwarded onwards to. These payment flows align with different roles in the network. Take the example of Alex routing a payment through your node to buy his daily Starbucks. It is likely that you will mostly see incoming payments from Alex, and outgoing payments to Starbucks along the channels you have open with them.

Understanding of these flows provides valuable information about how your node is operating. If you have a mostly inbound peer, like Alex purchasing his coffee, who is regularly sending payments with good fees your way, it is in your interest to maintain your incoming capacity over that channel by looping out or performing a circular rebalance. By contrast, there is little to gain by making an effort to maintain incoming capacity from Starbucks, who are mostly receiving forwards from your node. You may, however, want to top up your outgoing liquidity with Starbucks though looping in or opening another channel.

We have added a revenue report to Faraday to help surface information about these flows in your node. These reports are provided from the perspective of a single channel, called the target channel, and provide an account of the volume and fees that each channel that the target channel has been involved in forwards with has produced. Fees in lightning accrue on the incoming channel, but this report has been designed to distribute them evenly between incoming and outgoing channels, since both parties are responsible for the payment being forwarded.

./frcli revenue --chan_points=edc6090987cfd1b71f0a3b32ed2bd4504d9e96e7dd65444522b4fcb0e6eedf39:1 "reports": [ { // The channel between Alex -> You. "target_channel": "edc6090987cfd1b71f0a3b32ed2bd4504d9e96e7dd65444522b4fcb0e6eedf39:1", "pair_reports": { // The channel between You -> Starbucks. "1a2f48953a3c8402d7afa9f16de84a2dc7491876dc0e0b7ed05fe30b565dd656:1": { "amount_outgoing_msat": "3570000", "fees_outgoing_msat": "7000", "amount_incoming_msat": "0", "fees_incoming_msat": "0" }, } }, ] }

Going forward

In the Lightning future we envision, node operation will be largely automated, with node management tools conveniently packaged with your node. A blissful future is one where your Lightning node ‘just works’ out of the box, and the burden of daily operations does not lie on individual node operators. While we are still in the early stages of development of the network, experimentation is required to determine effective strategies to discover what ‘just works’.

Our hope with this tool is that it will assist node operators in identifying non-productive channels and more efficiently allocating their capital. Eliminating unusable channels will benefit individual operators and the network as a whole. Faraday’s source code is available today, and any issues or feature requests can be submitted using the Github issues page.