summaryrefslogtreecommitdiff
path: root/net/sched/sch_cake.c
AgeCommit message (Collapse)Author
2026-08-22net/sched: sch_cake: fix autorate reconfiguration throttlingGiuseppe Piscitelli
CAKE's autorate-ingress path intends to limit shaper reconfiguration to once per 250 ms, but last_reconfig_time is only checked and never updated. Since the field stays zero, every qualifying capacity-estimate window can call cake_reconfigure(), causing avoidable rate churn and scheduler work under bursty traffic. Store the current timestamp when autorate actually reconfigures the qdisc so the guard enforces the intended interval. Fixes: 7298de9cd725 ("sch_cake: Add ingress mode") Signed-off-by: Giuseppe Piscitelli <ooonea@gmail.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260820154503.892214-1-ooonea@gmail.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-08-03net/sched: sch_cake: drop WARN_ON(1) for malformed packets in ACK filterToke Høiland-Jørgensen
The sch_cake ACK filter parses packets to find the TCP header and filter duplicated ACKs if the flow is backlogged. The parsing code contains a WARN_ON(1) which can be triggered by a malformed IP header in certain cases. Depending on the system configuration, this leads either to either spamming dmesg with warnings, or a panic if panic_on_warn is set. The code already correctly skips the offending packet in the branch that triggers the warning, so the WARN_ON itself doesn't really serve any purpose. So just drop it altogether to avoid the inconvenient side effects. Fixes: 8b7138814f29 ("sch_cake: Add optional ACK filter") Reported-by: Zhiling Zou <zhilinz@nebusec.ai> Reported-by: Ren Wei <enjou1224z@gmail.com> Signed-off-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260729191417.45665-1-toke@toke.dk Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-07-24net/sched: sch_cake: skip clearing unused tins during rate adjustmentJonas Köppeler
When cake_configure_rates() is called from the dequeue path with rate_adjust=true, it only needs to update the rate parameters. The loop that clears the unused tins is both unnecessary and harmful in this path: - cake_clear_tin() overwrites q->cur_tin and q->cur_flow, which are actively used by cake_dequeue(), corrupting the dequeue state. - iterating over the unused tins and their internal queues to purge packets adds needless overhead to the hot path. Skip the entire loop when rate_adjust is set, as neither cake_clear_tin() nor the mtu_time update are needed when only the rate changes. The clearing loop runs on every rate adjustment from the dequeue path, clearing (max_tins - cur_tins) tins each time, so the cost grows the fewer tins the configured mode actually uses. Testing cake_mq over veth (8 rx/tx queues, 2 Gbit limit) with flent's [1] rrul and tcp_nup tests and 32 TCP upstreams shows a large drop in loaded latency and a throughput gain, restoring behaviour to pre-15c2715a5264 levels: +------------+------+------+-------+-------+---------+ | kernel | mode | test | base | load | tput | | | | | (ms) | (ms) | (Mbit) | +------------+------+------+-------+-------+---------+ | net-next | be | rrul | 0.810 | 11.78 | 1469.67 | | net-next | be | nup | 0.637 | 85.71 | 1243.15 | | net-next | ds3 | rrul | 0.397 | 15.28 | 1770.06 | | net-next | ds3 | nup | 0.351 | 15.98 | 1799.39 | +------------+------+------+-------+-------+---------+ | patched | be | rrul | 0.092 | 0.56 | 1873.40 | | patched | be | nup | 0.109 | 1.82 | 1869.12 | | patched | ds3 | rrul | 0.097 | 0.98 | 1866.10 | | patched | ds3 | nup | 0.101 | 0.51 | 1861.79 | +------------+------+------+-------+-------+---------+ The same trend holds on real hardware (IPQ8074A, 4 rx/tx queues, OpenWrt): in besteffort mode the tcp_nup loaded latency drops from ~470 ms to ~4 ms. [1] https://flent.org Fixes: 15c2715a5264 ("net/sched: sch_cake: fixup cake_mq rate adjustment for diffserv config") Signed-off-by: Jonas Köppeler <j.koeppeler@tu-berlin.de> Tested-by: Mike Pham <mikepham4321@gmail.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260720-sch_cake-skip-clearing-tins-v2-1-e6a8b0275c73@tu-berlin.de Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-07-20net/sched: Handle TC_ACT_REDIRECT from qdisc filter chainsJamal Hadi Salim
When a TC filter attached to a qdisc filter chain returns TC_ACT_REDIRECT (ex: via an eBPF program calling bpf_redirect() or an act_bpf action), the redirect was silently lost i.e no qdisc classify function handled TC_ACT_REDIRECT, so the packet fell through the switch and was enqueued normally instead of being redirected. This has been broken since bpf_redirect() was introduced for TC in commit 27b29f63058d ("bpf: add bpf_redirect() helper"). We got lucky for a long time because bpf_net_context was a per-CPU variable that was always available. commit 401cb7dae813 ("net: Reference bpf_redirect_info via task_struct on PREEMPT_RT.") turned bpf_net_context into a task_struct member that is only set up by explicit callers. Without a caller setting it up, bpf_redirect() itself crashes with a NULL pointer dereference in bpf_net_ctx_get_ri(). However, even with bpf_net_context available, TC_ACT_REDIRECT from qdisc filter chains cannot be honored without adding skb_do_redirect() calls to every qdisc classify function, which would require changes across net/sched/. Isolate it to ebpf core where it belongs. Instead, add a tcf_classify_qdisc() inline helper in pkt_cls.h, as a wrapper around tcf_classify() for use by qdisc classify functions and tcf_qevent_handle(). When the classify verdict is TC_ACT_REDIRECT, the wrapper converts it to TC_ACT_SHOT, dropping the packet rather than letting it continue silently. Dropping is preferred over letting the packet through because the user immediately sees packet loss. Silently passing the packet through would hide the problem and leave the user wondering why their redirect is not working. The clsact fast path, tc_run() continues to call tcf_classify() directly and is unaffected: TC_ACT_REDIRECT is returned as-is and handled by sch_handle_egress/ingress() calling skb_do_redirect() as before. Fixes: 27b29f63058d ("bpf: add bpf_redirect() helper") Fixes: 401cb7dae813 ("net: Reference bpf_redirect_info via task_struct on PREEMPT_RT.") Tested-by: Victor Nogueira <victor@mojatatu.com> Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Link: https://patch.msgid.link/20260706185609.330006-3-daniel@iogearbox.net Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-07-08net/sched: cake: reject overhead values that underflow lengthSamuel Moelius
CAKE accepts signed overhead values and stores them in an s16, but the adjusted packet length calculation uses unsigned arithmetic. A negative effective length can therefore wrap to a large value. Such configurations make rate accounting depend on integer wraparound rather than on the packet size userspace intended to model. A static netlink lower bound is not enough because packets reaching CAKE can be smaller than any reasonable manual-overhead allowance. Fold the signed overhead adjustment into the existing datapath MPU clamp so negative adjusted lengths are clamped before link-layer framing adjustments. Fixes: a729b7f0bd5b ("sch_cake: Add overhead compensation support to the rate shaper") Assisted-by: Codex:gpt-5.5-cyber-preview Signed-off-by: Samuel Moelius <sam.moelius@trailofbits.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260702000758.297407.e5c888d9d99d.cake-overhead-underflow@trailofbits.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2026-05-11net/sched: annotate data-races around sch->qstats.backlogEric Dumazet
Add qstats_backlog_sub() and qstats_backlog_add() helpers and use them instead of open-coding them. These helpers use WRITE_ONCE() to prevent store-tearing. Also use WRITE_ONCE() in fq_reset() and qdisc_reset() when sch->qstats.backlog is cleared. Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260510091455.4039245-4-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-05-11net/sched: add qdisc_qlen_inc() and qdisc_qlen_dec()Eric Dumazet
Helpers to increment or decrement sch->q.qlen, with appropriate WRITE_ONCE() to prevent store tearing. Add other WRITE_ONCE() when sch->q.qlen is changed. Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260510091455.4039245-3-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-05-07Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netJakub Kicinski
Cross-merge networking fixes after downstream PR (net-7.1-rc3). Conflicts: net/ipv4/igmp.c 726fa7da2d8c ("ipv4: igmp: get rid of IGMPV3_{QQIC,MRC} and simplify calculation") c6bebaa744f7 ("ipv4: igmp: annotate data-races in igmp_heard_query()") https://lore.kernel.org/a7365e4873340f7a5e30411207de3bf9@kernel.org Adjacent changes: net/psp/psp_main.c 30cb24f97d44 ("psp: strip variable-length PSP header in psp_dev_rcv()") c2b22277ad89 ("psp: validate IPv4 header fields in psp_dev_rcv()") net/sched/sch_fq_codel.c f83e07b29246 ("net/sched: sch_fq_codel: annotate data-races from fq_codel_dump_class_stats()") 3f3aa77ff1c8 ("net/sched: add qstats_cpu_drop_inc() helper") net/wireless/pmsr.c 0f3c0a197309 ("wifi: nl80211: fix NL80211_PMSR_FTM_REQ_ATTR_FTMS_PER_BURST usage") 410aa47fd9d3 ("wifi: cfg80211: allow suppressing FTM result reporting for PD requests") Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-05-04net/sched: add qstats_cpu_drop_inc() helperEric Dumazet
1) Using this_cpu_inc() is better than going through this_cpu_ptr(): - Single instruction on x86. - Store tearing prevention. 2) Change tcf_action_update_stats() to use this_cpu_add(). 3) Add WRITE_ONCE() to __qdisc_qstats_drop() and qstats_drop_inc() in preparation for lockless "tc qdisc show". $ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 3/17 up/down: 72/-216 (-144) Function old new delta dualpi2_enqueue_skb 462 511 +49 tcf_ife_act 1061 1077 +16 taprio_enqueue 613 620 +7 codel_qdisc_enqueue 149 143 -6 tcf_vlan_act 684 676 -8 tcf_skbedit_act 626 618 -8 tcf_police_act 725 717 -8 tcf_mpls_act 1297 1289 -8 tcf_gate_act 310 302 -8 tcf_gact_act 222 214 -8 tcf_csum_act 2438 2430 -8 tcf_bpf_act 709 701 -8 tcf_action_update_stats 124 115 -9 pie_qdisc_enqueue 865 856 -9 pfifo_enqueue 116 107 -9 choke_enqueue 2069 2059 -10 plug_enqueue 139 128 -11 bfifo_enqueue 121 110 -11 tcf_nat_act 1501 1489 -12 gred_enqueue 1743 1668 -75 Total: Before=24388609, After=24388465, chg -0.00% Signed-off-by: Eric Dumazet <edumazet@google.com> Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Link: https://patch.msgid.link/20260501135916.2566766-1-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-05-02net/sched: sch_cake: annotate data-races in cake_dump_class_stats (II)Eric Dumazet
cake_dump_class_stats() runs without qdisc spinlock being held. In this second patch, I add READ_ONCE()/WRITE_ONCE() annotations for: - flow->deficit - flow->cvars.dropping - flow->cvars.count - flow->cvars.p_drop - flow->cvars.blue_timer - flow->cvars.drop_next Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Reviewed-by: Simon Horman <horms@kernel.org> Link: https://patch.msgid.link/20260430061610.3503483-3-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-05-02net/sched: sch_cake: annotate data-races in cake_dump_class_stats (I)Eric Dumazet
cake_dump_class_stats() runs without qdisc spinlock being held. In this first patch, I add READ_ONCE()/WRITE_ONCE() annotations for: - flow->head - flow->dropped - b->backlogs[] Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Reviewed-by: Simon Horman <horms@kernel.org> Link: https://patch.msgid.link/20260430061610.3503483-2-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-28net/sched: sch_cake: annotate data-races in cake_dump_stats() (V)Eric Dumazet
cake_dump_stats() runs without qdisc spinlock being held. In this final patch, I add READ_ONCE()/WRITE_ONCE() annotations for cparams.target and cparams.interval. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: "Toke Høiland-Jørgensen" <toke@toke.dk> Link: https://patch.msgid.link/20260427083606.459355-6-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-28net/sched: sch_cake: annotate data-races in cake_dump_stats() (IV)Eric Dumazet
cake_dump_stats() runs without qdisc spinlock being held. In this fourth patch, I add READ_ONCE()/WRITE_ONCE() annotations for the following fields: - avg_peak_bandwidth - buffer_limit - buffer_max_used - avg_netoff - max_netlen - max_adjlen - min_netlen - min_adjlen - active_queues - tin_rate_bps - bytes - tin_backlog Other annotations are added in following patch, to ease code review. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260427083606.459355-5-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-28net/sched: sch_cake: annotate data-races in cake_dump_stats() (III)Eric Dumazet
cake_dump_stats() runs without qdisc spinlock being held. In this third patch, I add READ_ONCE()/WRITE_ONCE() annotations for the following fields: - packets - tin_dropped - tin_ecn_mark - ack_drops - peak_delay - avge_delay - base_delay Other annotations are added in following patches, to ease code review. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: "Toke Høiland-Jørgensen" <toke@toke.dk> Link: https://patch.msgid.link/20260427083606.459355-4-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-28net/sched: sch_cake: annotate data-races in cake_dump_stats() (II)Eric Dumazet
cake_dump_stats() runs without qdisc spinlock being held. In this second patch, I add READ_ONCE()/WRITE_ONCE() annotations for the following fields: - bulk_flow_count - unresponsive_flow_count - max_skblen - flow_quantum Other annotations are added in following patches, to ease code review. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: "Toke Høiland-Jørgensen" <toke@toke.dk> Link: https://patch.msgid.link/20260427083606.459355-3-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-28net/sched: sch_cake: annotate data-races in cake_dump_stats() (I)Eric Dumazet
cake_dump_stats() runs without qdisc spinlock being held. In this first patch, I add READ_ONCE()/WRITE_ONCE() annotations for the following fields: - way_hits - way_misses - way_collisions - sparse_flow_count - decaying_flow_count Other annotations are added in following patches, to ease code review. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: Eric Dumazet <edumazet@google.com> Acked-by: "Toke Høiland-Jørgensen" <toke@toke.dk> Link: https://patch.msgid.link/20260427083606.459355-2-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-04-16net/sched: sch_cake: fix NAT destination port not being updated in ↵Dudu Lu
cake_update_flowkeys cake_update_flowkeys() is supposed to update the flow dissector keys with the NAT-translated addresses and ports from conntrack, so that CAKE's per-flow fairness correctly identifies post-NAT flows as belonging to the same connection. For the source port, this works correctly: keys->ports.src = port; But for the destination port, the assignment is reversed: port = keys->ports.dst; This means the NAT destination port is never updated in the flow keys. As a result, when multiple connections are NATed to the same destination, CAKE treats them as separate flows because the original (pre-NAT) destination ports differ. This breaks CAKE's NAT-aware flow isolation when using the "nat" mode. The bug was introduced in commit b0c19ed6088a ("sch_cake: Take advantage of skb->hash where appropriate") which refactored the original direct assignment into a compare-and-conditionally-update pattern, but wrote the destination port update backwards. Fix by reversing the assignment direction to match the source port pattern. Fixes: b0c19ed6088a ("sch_cake: Take advantage of skb->hash where appropriate") Signed-off-by: Dudu Lu <phx0fer@gmail.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260413110041.44704-1-phx0fer@gmail.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2026-03-05Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netJakub Kicinski
Cross-merge networking fixes after downstream PR (net-7.0-rc3). No conflicts. Adjacent changes: net/netfilter/nft_set_rbtree.c fb7fb4016300 ("netfilter: nf_tables: clone set on flush only") 3aea466a4399 ("netfilter: nft_set_rbtree: don't disable bh when acquiring tree lock") Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-02-28net: sched: rename QDISC_DROP_CAKE_FLOOD to QDISC_DROP_FLOOD_PROTECTIONJesper Dangaard Brouer
Rename QDISC_DROP_CAKE_FLOOD to QDISC_DROP_FLOOD_PROTECTION to use a generic name without embedding the qdisc name. This follows the principle that drop reasons should describe the drop mechanism rather than being tied to a specific qdisc implementation. The flood protection drop reason is used by qdiscs implementing probabilistic drop algorithms (like BLUE) that detect unresponsive flows indicating potential DoS or flood attacks. CAKE uses this via its Cobalt AQM component. Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org> Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/177211347537.3011628.13759059534638729639.stgit@firesoul Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-02-28net: sched: introduce qdisc-specific drop reason tracingJesper Dangaard Brouer
Create new enum qdisc_drop_reason and trace_qdisc_drop tracepoint for qdisc layer drop diagnostics with direct qdisc context visibility. The new tracepoint includes qdisc handle, parent, kind (name), and device information. Existing SKB_DROP_REASON_QDISC_DROP is retained for backwards compatibility via kfree_skb_reason(). Convert qdiscs with drop reasons to use the new infrastructure. Change CAKE's cobalt_should_drop() return type from enum skb_drop_reason to enum qdisc_drop_reason to fix implicit enum conversion warnings. Use QDISC_DROP_UNSPEC as the 'not dropped' sentinel instead of SKB_NOT_DROPPED_YET. Both have the same compiled value (0), so the comparison logic remains semantically equivalent. Signed-off-by: Jesper Dangaard Brouer <hawk@kernel.org> Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/177211345275.3011628.1974310302645218067.stgit@firesoul Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-02-27net/sched: sch_cake: fixup cake_mq rate adjustment for diffserv configJonas Köppeler
cake_mq's rate adjustment during the sync periods did not adjust the rates for every tin in a diffserv config. This lead to inconsistencies of rates between the tins. Fix this by setting the rates for all tins during synchronization. Fixes: 1bddd758bac2 ("net/sched: sch_cake: share shaper state across sub-instances of cake_mq") Signed-off-by: Jonas Köppeler <j.koeppeler@tu-berlin.de> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260226-cake-mq-skip-sync-bandwidth-unlimited-v1-2-01830bb4db87@tu-berlin.de Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-02-27net/sched: sch_cake: avoid sync overhead when unlimitedJonas Köppeler
Skip inter-instance sync when no rate limit is configured, as it serves no purpose and only adds overhead. Fixes: 1bddd758bac2 ("net/sched: sch_cake: share shaper state across sub-instances of cake_mq") Signed-off-by: Jonas Köppeler <j.koeppeler@tu-berlin.de> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20260226-cake-mq-skip-sync-bandwidth-unlimited-v1-1-01830bb4db87@tu-berlin.de Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-02-21Convert more 'alloc_obj' cases to default GFP_KERNEL argumentsLinus Torvalds
This converts some of the visually simpler cases that have been split over multiple lines. I only did the ones that are easy to verify the resulting diff by having just that final GFP_KERNEL argument on the next line. Somebody should probably do a proper coccinelle script for this, but for me the trivial script actually resulted in an assertion failure in the middle of the script. I probably had made it a bit _too_ trivial. So after fighting that far a while I decided to just do some of the syntactically simpler cases with variations of the previous 'sed' scripts. The more syntactically complex multi-line cases would mostly really want whitespace cleanup anyway. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2026-02-21treewide: Replace kmalloc with kmalloc_obj for non-scalar typesKees Cook
This is the result of running the Coccinelle script from scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to avoid scalar types (which need careful case-by-case checking), and instead replace kmalloc-family calls that allocate struct or union object instances: Single allocations: kmalloc(sizeof(TYPE), ...) are replaced with: kmalloc_obj(TYPE, ...) Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...) are replaced with: kmalloc_objs(TYPE, COUNT, ...) Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...) are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...) (where TYPE may also be *VAR) The resulting allocations no longer return "void *", instead returning "TYPE *". Signed-off-by: Kees Cook <kees@kernel.org>
2026-02-06net/ipv6: Introduce payload_len helpersAlice Mikityanska
The next commits will transition away from using the hop-by-hop extension header to encode packet length for BIG TCP. Add wrappers around ip6->payload_len that return the actual value if it's non-zero, and calculate it from skb->len if payload_len is set to zero (and a symmetrical setter). The new helpers are used wherever the surrounding code supports the hop-by-hop jumbo header for BIG TCP IPv6, or the corresponding IPv4 code uses skb_ip_totlen (e.g., in include/net/netfilter/nf_tables_ipv6.h). No behavioral change in this commit. Signed-off-by: Alice Mikityanska <alice@isovalent.com> Acked-by: Paolo Abeni <pabeni@redhat.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Link: https://patch.msgid.link/20260205133925.526371-2-alice.kernel@fastmail.im Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-01-17net/sched: cake: avoid separate allocation of struct cake_sched_configToke Høiland-Jørgensen
Paolo pointed out that we can avoid separately allocating struct cake_sched_config even in the non-mq case, by embedding it into struct cake_sched_data. This reduces the complexity of the logic that swaps the pointers and frees the old value, at the cost of adding 56 bytes to the latter. Since cake_sched_data is already almost 17k bytes, this seems like a reasonable tradeoff. Suggested-by: Paolo Abeni <pabeni@redhat.com> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Fixes: bc0ce2bad36c ("net/sched: sch_cake: Factor out config variables into separate struct") Link: https://patch.msgid.link/20260113143157.2581680-1-toke@redhat.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2026-01-13net/sched: sch_cake: share shaper state across sub-instances of cake_mqJonas Köppeler
This commit adds shared shaper state across the cake instances beneath a cake_mq qdisc. It works by periodically tracking the number of active instances, and scaling the configured rate by the number of active queues. The scan is lockless and simply reads the qlen and the last_active state variable of each of the instances configured beneath the parent cake_mq instance. Locking is not required since the values are only updated by the owning instance, and eventual consistency is sufficient for the purpose of estimating the number of active queues. The interval for scanning the number of active queues is set to 200 us. We found this to be a good tradeoff between overhead and response time. For a detailed analysis of this aspect see the Netdevconf talk: https://netdevconf.info/0x19/docs/netdev-0x19-paper16-talk-paper.pdf Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Signed-off-by: Jonas Köppeler <j.koeppeler@tu-berlin.de> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20260109-mq-cake-sub-qdisc-v8-5-8d613fece5d8@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2026-01-13net/sched: sch_cake: Share config across cake_mq sub-qdiscsToke Høiland-Jørgensen
This adds support for configuring the cake_mq instance directly, sharing the config across the cake sub-qdiscs. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20260109-mq-cake-sub-qdisc-v8-4-8d613fece5d8@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2026-01-13net/sched: sch_cake: Add cake_mq qdisc for using cake on mq devicesToke Høiland-Jørgensen
Add a cake_mq qdisc which installs cake instances on each hardware queue on a multi-queue device. This is just a copy of sch_mq that installs cake instead of the default qdisc on each queue. Subsequent commits will add sharing of the config between cake instances, as well as a multi-queue aware shaper algorithm. Reviewed-by: Willem de Bruijn <willemb@google.com> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20260109-mq-cake-sub-qdisc-v8-3-8d613fece5d8@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2026-01-13net/sched: sch_cake: Factor out config variables into separate structToke Høiland-Jørgensen
Factor out all the user-configurable variables into a separate struct and embed it into struct cake_sched_data. This is done in preparation for sharing the configuration across multiple instances of cake in an mq setup. No functional change is intended with this patch. Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Reviewed-by: Willem de Bruijn <willemb@google.com> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20260109-mq-cake-sub-qdisc-v8-2-8d613fece5d8@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-12-02Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netJakub Kicinski
Merge in late fixes in preparation for the net-next PR. Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-12-02net/sched: sch_cake: Fix incorrect qlen reduction in cake_dropXiang Mei
In cake_drop(), qdisc_tree_reduce_backlog() is used to update the qlen and backlog of the qdisc hierarchy. Its caller, cake_enqueue(), assumes that the parent qdisc will enqueue the current packet. However, this assumption breaks when cake_enqueue() returns NET_XMIT_CN: the parent qdisc stops enqueuing current packet, leaving the tree qlen/backlog accounting inconsistent. This mismatch can lead to a NULL dereference (e.g., when the parent Qdisc is qfq_qdisc). This patch computes the qlen/backlog delta in a more robust way by observing the difference before and after the series of cake_drop() calls, and then compensates the qdisc tree accounting if cake_enqueue() returns NET_XMIT_CN. To ensure correct compensation when ACK thinning is enabled, a new variable is introduced to keep qlen unchanged. Fixes: 15de71d06a40 ("net/sched: Make cake_enqueue return NET_XMIT_CN when past buffer_limit") Signed-off-by: Xiang Mei <xmei5@asu.edu> Reviewed-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://patch.msgid.link/20251128001415.377823-1-xmei5@asu.edu Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-11-25net_sched: use qdisc_dequeue_drop() in cake, codel, fq_codelEric Dumazet
cake, codel and fq_codel can drop many packets from dequeue(). Use qdisc_dequeue_drop() so that the freeing can happen outside of the qdisc spinlock scope. Add TCQ_F_DEQUEUE_DROPS to sch->flags. Signed-off-by: Eric Dumazet <edumazet@google.com> Link: https://patch.msgid.link/20251121083256.674562-15-edumazet@google.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-11-25net_sched: cake: use qdisc_pkt_segs()Eric Dumazet
Use new qdisc_pkt_segs() to avoid a cache line miss in cake_enqueue() for non GSO packets. cake_overhead() does not have to recompute it. Signed-off-by: Eric Dumazet <edumazet@google.com> Link: https://patch.msgid.link/20251121083256.674562-7-edumazet@google.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-11-25net_sched: use qdisc_skb_cb(skb)->pkt_segs in bstats_update()Eric Dumazet
Avoid up to two cache line misses in qdisc dequeue() to fetch skb_shinfo(skb)->gso_segs/gso_size while qdisc spinlock is held. This gives a 5 % improvement in a TX intensive workload. Signed-off-by: Eric Dumazet <edumazet@google.com> Link: https://patch.msgid.link/20251121083256.674562-6-edumazet@google.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-11-25net_sched: initialize qdisc_skb_cb(skb)->pkt_segs in qdisc_pkt_len_init()Eric Dumazet
qdisc_pkt_len_init() is currently initalizing qdisc_skb_cb(skb)->pkt_len. Add qdisc_skb_cb(skb)->pkt_segs initialization and rename this function to qdisc_pkt_len_segs_init(). Signed-off-by: Eric Dumazet <edumazet@google.com> Link: https://patch.msgid.link/20251121083256.674562-4-edumazet@google.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2025-08-20net/sched: Make cake_enqueue return NET_XMIT_CN when past buffer_limitWilliam Liu
The following setup can trigger a WARNING in htb_activate due to the condition: !cl->leaf.q->q.qlen tc qdisc del dev lo root tc qdisc add dev lo root handle 1: htb default 1 tc class add dev lo parent 1: classid 1:1 \ htb rate 64bit tc qdisc add dev lo parent 1:1 handle f: \ cake memlimit 1b ping -I lo -f -c1 -s64 -W0.001 127.0.0.1 This is because the low memlimit leads to a low buffer_limit, which causes packet dropping. However, cake_enqueue still returns NET_XMIT_SUCCESS, causing htb_enqueue to call htb_activate with an empty child qdisc. We should return NET_XMIT_CN when packets are dropped from the same tin and flow. I do not believe return value of NET_XMIT_CN is necessary for packet drops in the case of ack filtering, as that is meant to optimize performance, not to signal congestion. Fixes: 046f6fd5daef ("sched: Add Common Applications Kept Enhanced (cake) qdisc") Signed-off-by: William Liu <will@willsroot.io> Reviewed-by: Savino Dicanosa <savy@syst3mfailure.io> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Link: https://patch.msgid.link/20250819033601.579821-1-will@willsroot.io Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-07-08net: account for encap headers in qdisc pkt lenFengyuan Gong
Refine qdisc_pkt_len_init to include headers up through the inner transport header when computing header size for encapsulations. Also refine net/sched/sch_cake.c borrowed from qdisc_pkt_len_init(). Signed-off-by: Fengyuan Gong <gfengyuan@google.com> Reviewed-by: Willem de Bruijn <willemb@google.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Acked-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20250702160741.1204919-1-gfengyuan@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-01-09Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/netJakub Kicinski
Cross-merge networking fixes after downstream PR (net-6.13-rc7). Conflicts: a42d71e322a8 ("net_sched: sch_cake: Add drop reasons") 737d4d91d35b ("sched: sch_cake: add bounds checks to host bulk flow fairness counts") Adjacent changes: drivers/net/ethernet/meta/fbnic/fbnic.h 3a856ab34726 ("eth: fbnic: add IRQ reuse support") 95978931d55f ("eth: fbnic: Revert "eth: fbnic: Add hardware monitoring support via HWMON interface"") Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2025-01-09sched: sch_cake: add bounds checks to host bulk flow fairness countsToke Høiland-Jørgensen
Even though we fixed a logic error in the commit cited below, syzbot still managed to trigger an underflow of the per-host bulk flow counters, leading to an out of bounds memory access. To avoid any such logic errors causing out of bounds memory accesses, this commit factors out all accesses to the per-host bulk flow counters to a series of helpers that perform bounds-checking before any increments and decrements. This also has the benefit of improving readability by moving the conditional checks for the flow mode into these helpers, instead of having them spread out throughout the code (which was the cause of the original logic error). As part of this change, the flow quantum calculation is consolidated into a helper function, which means that the dithering applied to the ost load scaling is now applied both in the DRR rotation and when a sparse flow's quantum is first initiated. The only user-visible effect of this is that the maximum packet size that can be sent while a flow stays sparse will now vary with +/- one byte in some cases. This should not make a noticeable difference in practice, and thus it's not worth complicating the code to preserve the old behaviour. Fixes: 546ea84d07e3 ("sched: sch_cake: fix bulk flow accounting logic for host fairness") Reported-by: syzbot+f63600d288bfb7057424@syzkaller.appspotmail.com Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Acked-by: Dave Taht <dave.taht@gmail.com> Link: https://patch.msgid.link/20250107120105.70685-1-toke@redhat.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-12-12net_sched: sch_cake: Add drop reasonsToke Høiland-Jørgensen
Add three qdisc-specific drop reasons and use them in sch_cake: 1) SKB_DROP_REASON_QDISC_OVERLIMIT Whenever the total queue limit for a qdisc instance is exceeded and a packet is dropped to make room. 2) SKB_DROP_REASON_QDISC_CONGESTED Whenever a packet is dropped by the qdisc AQM algorithm because congestion is detected. 3) SKB_DROP_REASON_CAKE_FLOOD Whenever a packet is dropped by the flood protection part of the CAKE AQM algorithm (BLUE). Also use the existing SKB_DROP_REASON_QUEUE_PURGE in cake_clear_tin(). Reasons show up as: perf record -a -e skb:kfree_skb sleep 1; perf script iperf3 665 [005] 848.656964: skb:kfree_skb: skbaddr=0xffff98168a333500 rx_sk=(nil) protocol=34525 location=__dev_queue_xmit+0x10f0 reason: QDISC_OVERLIMIT swapper 0 [001] 909.166055: skb:kfree_skb: skbaddr=0xffff98168280cee0 rx_sk=(nil) protocol=34525 location=cake_dequeue+0x5ef reason: QDISC_CONGESTED Reviewed-by: Eric Dumazet <edumazet@google.com> Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Acked-by: Dave Taht <dave.taht@gmail.com> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20241211-cake-drop-reason-v2-1-920afadf4d1b@redhat.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-12-04net: sched: fix ordering of qlen adjustmentLion Ackermann
Changes to sch->q.qlen around qdisc_tree_reduce_backlog() need to happen _before_ a call to said function because otherwise it may fail to notify parent qdiscs when the child is about to become empty. Signed-off-by: Lion Ackermann <nnamrec@gmail.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Signed-off-by: David S. Miller <davem@davemloft.net>
2024-09-10sch_cake: constify inverse square root cacheDave Taht
sch_cake uses a cache of the first 16 values of the inverse square root calculation for the Cobalt AQM to save some cycles on the fast path. This cache is populated when the qdisc is first loaded, but there's really no reason why it can't just be pre-populated. So change it to be pre-populated with constants, which also makes it possible to constify it. This gives a modest space saving for the module (not counting debug data): .text: -224 bytes .rodata: +80 bytes .bss: -64 bytes Total: -192 bytes Signed-off-by: Dave Taht <dave.taht@gmail.com> [ fixed up comment, rewrote commit message ] Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20240909091630.22177-1-toke@redhat.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-09-05sched: sch_cake: fix bulk flow accounting logic for host fairnessToke Høiland-Jørgensen
In sch_cake, we keep track of the count of active bulk flows per host, when running in dst/src host fairness mode, which is used as the round-robin weight when iterating through flows. The count of active bulk flows is updated whenever a flow changes state. This has a peculiar interaction with the hash collision handling: when a hash collision occurs (after the set-associative hashing), the state of the hash bucket is simply updated to match the new packet that collided, and if host fairness is enabled, that also means assigning new per-host state to the flow. For this reason, the bulk flow counters of the host(s) assigned to the flow are decremented, before new state is assigned (and the counters, which may not belong to the same host anymore, are incremented again). Back when this code was introduced, the host fairness mode was always enabled, so the decrement was unconditional. When the configuration flags were introduced the *increment* was made conditional, but the *decrement* was not. Which of course can lead to a spurious decrement (and associated wrap-around to U16_MAX). AFAICT, when host fairness is disabled, the decrement and wrap-around happens as soon as a hash collision occurs (which is not that common in itself, due to the set-associative hashing). However, in most cases this is harmless, as the value is only used when host fairness mode is enabled. So in order to trigger an array overflow, sch_cake has to first be configured with host fairness disabled, and while running in this mode, a hash collision has to occur to cause the overflow. Then, the qdisc has to be reconfigured to enable host fairness, which leads to the array out-of-bounds because the wrapped-around value is retained and used as an array index. It seems that syzbot managed to trigger this, which is quite impressive in its own right. This patch fixes the issue by introducing the same conditional check on decrement as is used on increment. The original bug predates the upstreaming of cake, but the commit listed in the Fixes tag touched that code, meaning that this patch won't apply before that. Fixes: 712639929912 ("sch_cake: Make the dual modes fairer") Reported-by: syzbot+7fe7b81d602cc1e6b94d@syzkaller.appspotmail.com Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20240903160846.20909-1-toke@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2024-04-19net_sched: cake: implement lockless cake_dump()Eric Dumazet
Instead of relying on RTNL, cake_dump() can use READ_ONCE() annotations, paired with WRITE_ONCE() ones in cake_change(). v2: addressed Simon feedback in V1: https://lore.kernel.org/netdev/20240417083549.GA3846178@kernel.org/ Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Toke Høiland-Jørgensen <toke@toke.dk> Reviewed-by: Simon Horman <horms@kernel.org> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Signed-off-by: David S. Miller <davem@davemloft.net>
2024-04-09net: sched: cake: Optimize the number of function calls and branches in heap ↵Kuan-Wei Chiu
construction When constructing a heap, heapify operations are required on all non-leaf nodes. Thus, determining the index of the first non-leaf node is crucial. In a heap, the left child's index of node i is 2 * i + 1 and the right child's index is 2 * i + 2. Node CAKE_MAX_TINS * CAKE_QUEUES / 2 has its left and right children at indexes CAKE_MAX_TINS * CAKE_QUEUES + 1 and CAKE_MAX_TINS * CAKE_QUEUES + 2, respectively, which are beyond the heap's range, indicating it as a leaf node. Conversely, node CAKE_MAX_TINS * CAKE_QUEUES / 2 - 1 has a left child at index CAKE_MAX_TINS * CAKE_QUEUES - 1, confirming its non-leaf status. The loop should start from it since it's not a leaf node. By starting the loop from CAKE_MAX_TINS * CAKE_QUEUES / 2 - 1, we minimize function calls and branch condition evaluations. This adjustment theoretically reduces two function calls (one for cake_heapify() and another for cake_heap_get_backlog()) and five branch evaluations (one for iterating all non-leaf nodes, one within cake_heapify()'s while loop, and three more within the while loop with if conditions). Signed-off-by: Kuan-Wei Chiu <visitorckw@gmail.com> Acked-by: Toke Høiland-Jørgensen <toke@toke.dk> Link: https://lore.kernel.org/r/20240408174716.751069-1-visitorckw@gmail.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2024-02-02net/sched: Add module aliases for cls_,sch_,act_ modulesMichal Koutný
No functional change intended, aliases will be used in followup commits. Note for backporters: you may need to add aliases also for modules that are already removed in mainline kernel but still in your version. Patches were generated with the help of Coccinelle scripts like: cat >scripts/coccinelle/misc/tcf_alias.cocci <<EOD virtual patch virtual report @ haskernel @ @@ @ tcf_has_kind depends on report && haskernel @ identifier ops; constant K; @@ static struct tcf_proto_ops ops = { .kind = K, ... }; +char module_alias = K; EOD /usr/bin/spatch -D report --cocci-file scripts/coccinelle/misc/tcf_alias.cocci \ --dir . \ -I ./arch/x86/include -I ./arch/x86/include/generated -I ./include \ -I ./arch/x86/include/uapi -I ./arch/x86/include/generated/uapi \ -I ./include/uapi -I ./include/generated/uapi \ --include ./include/linux/compiler-version.h --include ./include/linux/kconfig.h \ --jobs 8 --chunksize 1 2>/dev/null | \ sed 's/char module_alias = "\([^"]*\)";/MODULE_ALIAS_NET_CLS("\1");/' And analogously for: static struct tc_action_ops ops = { .kind = K, static struct Qdisc_ops ops = { .id = K, (Someone familiar would be able to fit those into one .cocci file without sed post processing.) Signed-off-by: Michal Koutný <mkoutny@suse.com> Acked-by: Jamal Hadi Salim <jhs@mojatatu.com> Reviewed-by: Jiri Pirko <jiri@nvidia.com> Link: https://lore.kernel.org/r/20240201130943.19536-3-mkoutny@suse.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-06-10net: move gso declarations and functions to their own filesEric Dumazet
Move declarations into include/net/gso.h and code into net/core/gso.c Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Stanislav Fomichev <sdf@google.com> Reviewed-by: Simon Horman <simon.horman@corigine.com> Reviewed-by: David Ahern <dsahern@kernel.org> Link: https://lore.kernel.org/r/20230608191738.3947077-1-edumazet@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-03-22sch_cake: do not use skb_mac_header() in cake_overhead()Eric Dumazet
We want to remove our use of skb_mac_header() in tx paths, eg remove skb_reset_mac_header() from __dev_queue_xmit(). Idea is that ndo_start_xmit() can get the mac header simply looking at skb->data. Signed-off-by: Eric Dumazet <edumazet@google.com> Reviewed-by: Simon Horman <simon.horman@corigine.com> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2023-02-01net: sched: use skb_ip_totlen and iph_totlenXin Long
There are 1 action and 1 qdisc that may process IPv4 TCP GSO packets and access iph->tot_len, replace them with skb_ip_totlen() and iph_totlen() accordingly. Note that we don't need to replace the one in tcf_csum_ipv4(), as it will return for TCP GSO packets in tcf_csum_ipv4_tcp(). Signed-off-by: Xin Long <lucien.xin@gmail.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
generated by cgit v1.2.3 (git 2.0.4) at 2026-09-17 23:43:04 +0000