You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 00:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 18, 2026 12:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3387540998783,2cyclictest0-21swapper/011:00:040
3387540998783,2cyclictest0-21swapper/011:00:030
338754099805,8cyclictest0-21swapper/011:15:030
338754099805,8cyclictest0-21swapper/011:15:020
3387540997929,46cyclictest3411548-21ssh09:49:570
3387552997727,18cyclictest3393720-21diskmemload09:30:012
3387552997621,18cyclictest3464439-21ssh11:46:432
3387552997520,19cyclictest3426105-21sh10:21:502
3387540997530,41cyclictest3456248-21ssh11:28:520
3387540997517,37cyclictest16850irq/134-800000009:12:070
3387552997415,21cyclictest3458913-21ssh11:34:552
3387552997414,16cyclictest3410862-21ssh09:48:132
3387540997320,49cyclictest3398537-21ssh09:20:100
3387552997216,14cyclictest3393723-21kworker/u8:409:32:322
338754099711,53cyclictest3467044-21ssh11:52:440
338754099711,53cyclictest3467044-21ssh11:52:430
3387552997017,17cyclictest3417237-21rm10:02:132
3387540997022,43cyclictest3458455-21ssh11:33:450
338754099691,9cyclictest3467602-21systemd-journal09:57:110
338754099691,64cyclictest0-21swapper/009:52:490
3387540996916,48cyclictest16850irq/134-800000011:07:270
3387552996818,16cyclictest3475274-21ssh12:10:362
3387552996815,19cyclictest3428836-21ssh10:27:592
3387552996815,19cyclictest3391936-21cat08:35:012
338754099681,63cyclictest3427697-21sshd10:25:120
3387552996715,15cyclictest3422413-21ssh10:13:492
3387552996714,19cyclictest3478032-21ssh12:16:552
3387552996714,15cyclictest3396996-21ssh09:17:002
338754099673,6cyclictest3461454-21cat11:40:030
3387540996722,27cyclictest3409498-21ssh09:44:570
3387540996718,32cyclictest3427160-21ssh10:24:280
3387540996713,38cyclictest16850irq/134-800000011:16:470
338754099671,14cyclictest3486231-21latency_hist12:35:020
3387540996710,6cyclictest16850irq/134-800000011:55:420
3387552996620,19cyclictest3415088-21ssh09:57:342
3387540996611,6cyclictest16850irq/134-800000012:19:460
338754099661,12cyclictest3487154-21sshd12:37:110
3387552996517,19cyclictest3455635-21sh11:27:202
3387552996515,15cyclictest3408529-21ssh09:42:352
3387552996512,15cyclictest3487783-21ssh12:38:432
338754099655,8cyclictest3390855-21munin-plugin-st08:15:010
338754099652,12cyclictest48-21kcompactd010:12:040
3387555996442,17cyclictest3467602-21systemd-journal09:15:003
338754099646,10cyclictest3418740-21ssh10:05:160
3387540996433,26cyclictest3474720-21sh12:09:500
338754099642,56cyclictest1-21systemd11:42:110
338754099642,56cyclictest1-21systemd11:42:110
338754099642,15cyclictest3392696-21munin-run08:50:020
338754099642,14cyclictest3389567-21munin-run07:50:000
3387552996318,15cyclictest3474182-21ssh12:08:302
3387552996316,14cyclictest3467417-21ssh11:53:342
3387552996316,14cyclictest3467417-21ssh11:53:332
338754099632,13cyclictest3393225-21munin-run09:00:010
3387540996321,36cyclictest16850irq/134-800000010:40:000
338754099631,11cyclictest3391141-21sendmail-msp08:20:000
338754099624,10cyclictest3405017-21modprobe09:35:000
3387540996240,19cyclictest16950irq/224-800000007:45:000
3387540996232,26cyclictest3476104-21rm12:12:480
3387540996230,28cyclictest3454243-21ssh11:24:330
3387540996222,35cyclictest12-21ksoftirqd/010:55:320
338754099622,14cyclictest364-21snmpd10:01:170
3387540996215,43cyclictest3429908-21latency_hist10:30:030
3387552996144,14cyclictest3453757-21ssh11:23:192
338754099612,13cyclictest3392159-21munin-run08:40:010
3387540996113,42cyclictest0-21swapper/008:00:000
338754099611,13cyclictest3391881-21sshd08:34:110
338754099611,10cyclictest3467602-21systemd-journal11:49:130
3387540996031,25cyclictest3482557-21sh12:26:520
338754099602,12cyclictest3472559-21systemd-run12:05:020
3387540996018,26cyclictest3438402-21ssh10:49:260
338754099601,39cyclictest3436559-21wc10:45:020
338754099601,13cyclictest3424764-21sshd10:19:110
338754099601,12cyclictest3392898-21sshd08:50:110
3387552995912,17cyclictest3469365-21ssh11:57:482
3387540995937,18cyclictest16950irq/224-800000010:51:210
3387540995912,26cyclictest16850irq/134-800000009:35:230
338755599589,45cyclictest3454629-21ssh11:25:023
3387552995844,11cyclictest3394207-21ssh09:10:342
3387552995835,19cyclictest3480791-21ssh12:23:072
3387552995815,13cyclictest3440895-21ssh10:54:592
338754099582,15cyclictest3401812-21ssh09:27:450
338754099581,17cyclictest3480731-21ssh12:22:590
3387552995736,17cyclictest3485498-21ssh12:33:342
3387552995736,17cyclictest3448941-21ssh11:12:322
3387552995736,17cyclictest3448941-21ssh11:12:322
3387552995736,17cyclictest3405986-21ssh09:36:502
3387552995736,17cyclictest3391365-21sshd08:21:112
3387552995736,17cyclictest3391365-21sshd08:21:112
3387552995713,13cyclictest3444838-21ssh11:03:352
3387552995713,13cyclictest3444838-21ssh11:03:352
3387540995721,32cyclictest16950irq/224-800000007:30:050
3387555995639,13cyclictest1-21systemd12:30:003
3387555995619,14cyclictest37-21ksoftirqd/308:35:013
3387552995634,18cyclictest3460397-21ssh11:38:052
338754099561,2cyclictest0-21swapper/009:18:320
338754099561,12cyclictest364-21snmpd07:10:470
3387555995530,19cyclictest275-21dbus-daemon09:55:013
3387552995542,10cyclictest3419723-21ssh10:07:522
3387552995539,13cyclictest3441575-21ssh10:56:052
3387552995539,12cyclictest3437633-21ssh10:47:332
3387552995538,14cyclictest3434550-21ssh10:40:212
3387552995535,16cyclictest3424741-21ssh10:19:082
3387552995534,17cyclictest3429916-21ssh10:30:042
338754599552,39cyclictest221rcuc/110:20:031
338754099552,7cyclictest3393725-21modprobe09:10:000
338754099552,15cyclictest3390350-21munin-run08:05:000
3387552995444,6cyclictest3391419-21kworker/2:008:40:002
3387552995439,12cyclictest3471797-21ssh12:03:092
3387552995436,14cyclictest3398982-21ssh09:21:212
3387552995433,17cyclictest3450597-21ssh11:16:012
3387552995433,17cyclictest3446643-21ssh11:07:232
3387552995411,14cyclictest3433339-21ssh10:38:032
338754099541,12cyclictest3391723-21latency_hist08:30:010
3387552995337,14cyclictest3463033-21ssh11:43:502
3387552995337,14cyclictest3463033-21ssh11:43:502
3387552995336,13cyclictest3482718-21ssh12:27:242
3387552995335,14cyclictest3412359-21ssh09:51:282
338754099531,12cyclictest3389782-21sshd07:50:110
3387555995233,14cyclictest1-21systemd11:55:023
3387555995233,14cyclictest1-21systemd11:55:013
3387555995233,13cyclictest3467602-21systemd-journal09:40:003
3387555995226,19cyclictest361rcuc/310:05:033
3387552995235,13cyclictest3393150-21sshd08:55:112
338754099522,9cyclictest3467602-21systemd-journal07:25:010
338754099522,13cyclictest3393434-21sshd09:01:110
338754099521,20cyclictest3388010-21munin-run07:20:010
3387555995119,23cyclictest37-21ksoftirqd/307:20:023
3387552995137,11cyclictest3392904-21sshd08:51:112
3387552995129,15cyclictest3390811-21latency_hist08:10:022
3387552995124,21cyclictest3389414-21cat07:45:032
338754099511,11cyclictest3387535-21cyclictest07:10:020
3387555995031,15cyclictest1-21systemd10:15:003
3387545995038,9cyclictest1-21systemd10:15:011
3387545995034,12cyclictest3405207-21sed09:35:021
3387545995031,14cyclictest1-21systemd12:10:011
3387545995031,13cyclictest3467602-21systemd-journal08:40:011
3387552994937,9cyclictest364-21snmpd08:40:182
3387552994937,9cyclictest3391628-21sshd08:29:112
3387552994937,9cyclictest3390866-21munin-run08:15:022
3387552994935,9cyclictest3393479-21munin-run09:05:012
338754099492,17cyclictest3391394-21munin-run08:25:010
338754099492,17cyclictest3391394-21munin-run08:25:000
338754099492,14cyclictest3389047-21systemctl07:40:000
3387555994835,10cyclictest3438298-21sshd10:49:113
3387552994833,12cyclictest3391068-21sshd08:15:112
3387545994834,11cyclictest3467602-21systemd-journal09:51:111
338754099482,13cyclictest1-21systemd07:30:000
3387540994820,24cyclictest12-21ksoftirqd/008:42:540
3387555994738,6cyclictest37-21ksoftirqd/312:00:423
338755599472,4cyclictest131rcu_preempt09:19:133
338755599471,5cyclictest131rcu_preempt11:00:013
3387552994739,5cyclictest3367066-21kworker/2:207:40:022
3387552994735,9cyclictest3467602-21systemd-journal07:20:262
3387552994735,9cyclictest3388745-21sshd07:30:052
3387552994734,9cyclictest48-21kcompactd008:01:272
3387552994733,9cyclictest48-21kcompactd007:19:502
3387552994733,10cyclictest3388543-21/usr/sbin/munin07:30:012
338754599473,3cyclictest131rcu_preempt09:18:261
3387545994727,14cyclictest3467602-21systemd-journal11:25:011
3387555994624,17cyclictest275-21dbus-daemon11:35:013
3387552994633,9cyclictest48-21kcompactd007:50:552
3387545994633,9cyclictest1-21systemd08:30:001
3387545994631,10cyclictest3390078-21systemctl08:00:001
3387545994627,13cyclictest3467602-21systemd-journal11:25:491
338754099461,10cyclictest3390615-21unin-run08:10:010
338755599457,5cyclictest131rcu_preempt11:43:003
338755599457,5cyclictest131rcu_preempt11:42:593
3387555994520,20cyclictest37-21ksoftirqd/311:14:293
3387555994520,20cyclictest37-21ksoftirqd/311:14:293
3387552994537,5cyclictest3392684-21munin-run08:50:002
3387552994537,4cyclictest3387741-21munin-run07:15:002
338754599454,3cyclictest131rcu_preempt12:00:021
3387545994527,12cyclictest0-21swapper/112:25:011
338755599443,3cyclictest131rcu_preempt10:34:403
3387555994428,13cyclictest3393201-21tr09:00:003
3387555994425,12cyclictest309-21in:imuxsock07:40:013
3387555994424,15cyclictest3467602-21systemd-journal08:50:003
3387552994414,26cyclictest3387558-21cut07:10:022
3387545994430,12cyclictest23-21ksoftirqd/109:20:521
3387545994430,10cyclictest1-21systemd09:05:011
3387555994332,7cyclictest3434177-21sh10:40:003
3387555994328,12cyclictest37-21ksoftirqd/309:34:243
3387552994333,6cyclictest3390117-21latency_hist08:00:012
3387552994332,8cyclictest48-21kcompactd007:48:082
3387545994331,9cyclictest23-21ksoftirqd/112:16:271
3387545994330,9cyclictest3467602-21systemd-journal07:10:121
3387545994329,10cyclictest23-21ksoftirqd/110:45:001
3387545994328,12cyclictest23-21ksoftirqd/112:30:011
3387545994328,10cyclictest1-21systemd11:50:001
3387545994326,12cyclictest23-21ksoftirqd/111:10:011
338754599432,3cyclictest131rcu_preempt12:38:231
338755599424,3cyclictest131rcu_preempt12:21:233
3387555994230,9cyclictest37-21ksoftirqd/307:55:003
338755599422,3cyclictest131rcu_preempt10:09:183
3387555994223,14cyclictest3461205-21systemctl11:40:003
3387555994218,4cyclictest37-21ksoftirqd/309:21:343
3387555994218,20cyclictest37-21ksoftirqd/310:25:003
3387552994235,4cyclictest3393727-21munin-run09:10:012
3387545994228,10cyclictest23-21ksoftirqd/109:10:211
3387545994227,9cyclictest275-21dbus-daemon11:55:021
3387545994227,9cyclictest275-21dbus-daemon11:55:011
3387545994227,10cyclictest3443208-21cron11:00:001
3387545994227,10cyclictest309-21in:imuxsock10:20:021
3387545994225,12cyclictest23-21ksoftirqd/107:40:011
3387555994129,9cyclictest3391114-21cron08:20:003
3387555994126,12cyclictest275-21dbus-daemon08:40:003
3387555994126,10cyclictest3446138-21sh11:06:113
338755599411,4cyclictest131rcu_preempt11:28:243
3387545994129,9cyclictest3388765-21sshd07:34:111
3387545994129,7cyclictest3452180-21cron11:20:011
3387545994128,7cyclictest3467602-21systemd-journal08:15:001
3387545994127,10cyclictest3467602-21systemd-journal09:40:001
3387545994127,10cyclictest3416127-21systemctl10:00:001
338754599412,6cyclictest131rcu_preempt09:50:001
3387545994126,11cyclictest3467602-21systemd-journal09:00:001
3387545994126,11cyclictest23-21ksoftirqd/110:00:081
3387545994125,11cyclictest3467602-21systemd-journal07:20:001
3387545994113,6cyclictest131rcu_preempt10:28:151
338755599404,6cyclictest131rcu_preempt12:20:003
3387555994031,5cyclictest3393720-21diskmemload11:46:533
3387555994027,8cyclictest3393033-21latency_hist08:55:023
3387555994026,10cyclictest0-21swapper/312:00:013
3387555994024,11cyclictest3467602-21systemd-journal10:20:013
3387555994019,5cyclictest37-21ksoftirqd/312:30:523
3387555994017,4cyclictest37-21ksoftirqd/312:35:243
338755599401,3cyclictest131rcu_preempt11:16:063
338754599408,19cyclictest221rcuc/109:45:021
3387545994031,6cyclictest3470584-21ssh12:00:091
3387545994027,9cyclictest309-21in:imuxsock08:35:001
3387545994027,10cyclictest23-21ksoftirqd/110:32:581
338755599394,3cyclictest131rcu_preempt11:00:233
338755599394,3cyclictest131rcu_preempt11:00:233
3387555993931,4cyclictest3389565-21munin-plugin-st07:50:003
3387555993927,10cyclictest37-21ksoftirqd/309:55:203
3387555993926,11cyclictest37-21ksoftirqd/309:50:023
3387555993925,10cyclictest37-21ksoftirqd/312:10:173
3387555993924,10cyclictest37-21ksoftirqd/308:30:003
3387555993913,6cyclictest131rcu_preempt12:05:083
338754599393,3cyclictest131rcu_preempt11:42:541
338754599393,3cyclictest131rcu_preempt11:42:531
3387545993928,9cyclictest23-21ksoftirqd/111:40:021
3387545993928,8cyclictest1-21systemd08:25:011
3387545993928,8cyclictest1-21systemd08:25:011
3387545993927,10cyclictest23-21ksoftirqd/110:09:171
3387545993924,11cyclictest275-21dbus-daemon10:40:011
3387545993917,16cyclictest275-21dbus-daemon11:15:011
3387545993917,16cyclictest275-21dbus-daemon11:15:011
338755599383,3cyclictest131rcu_preempt09:25:043
3387555993829,7cyclictest37-21ksoftirqd/310:50:483
3387555993828,7cyclictest37-21ksoftirqd/308:05:023
3387555993825,8cyclictest3388770-21cron07:35:013
3387555993823,11cyclictest3467602-21systemd-journal07:30:013
3387555993823,10cyclictest3391389-21sh08:25:003
3387555993823,10cyclictest3391389-21sh08:25:003
3387555993819,15cyclictest37-21ksoftirqd/308:45:013
338754599383,3cyclictest131rcu_preempt11:34:241
3387545993829,5cyclictest309-21in:imuxsock12:15:011
3387545993827,9cyclictest23-21ksoftirqd/107:25:021
3387545993827,8cyclictest3390579-21sshd08:08:111
3387545993826,5cyclictest23-21ksoftirqd/112:35:011
3387545993826,10cyclictest23-21ksoftirqd/109:25:311
338755599374,4cyclictest131rcu_preempt09:05:013
338755599373,5cyclictest131rcu_preempt09:45:013
3387555993730,4cyclictest3390071-21latency_hist08:00:013
3387555993721,13cyclictest37-21ksoftirqd/310:43:563
3387555993717,15cyclictest275-21dbus-daemon07:15:003
3387555993713,6cyclictest131rcu_preempt10:26:053
3387545993726,9cyclictest23-21ksoftirqd/111:05:001
3387545993726,9cyclictest23-21ksoftirqd/111:05:001
3387545993726,9cyclictest23-21ksoftirqd/110:51:321
3387545993723,10cyclictest275-21dbus-daemon08:05:001
3387545993721,11cyclictest3467602-21systemd-journal08:20:011
3387555993627,7cyclictest37-21ksoftirqd/307:45:013
3387555993625,8cyclictest37-21ksoftirqd/308:10:003
3387545993625,7cyclictest309-21in:imuxsock09:10:021
3387545993622,11cyclictest23-21ksoftirqd/108:55:001
3387555993515,12cyclictest3388002-21cron07:20:003
3387545993524,8cyclictest23-21ksoftirqd/107:28:111
3387545993523,10cyclictest23-21ksoftirqd/110:48:111
3387555993423,8cyclictest3393742-21systemctl09:10:013
3387545993424,7cyclictest3387546-21latency_hist07:10:021
3387555993322,6cyclictest37-21ksoftirqd/308:15:013
3387545993321,8cyclictest23-21ksoftirqd/107:47:111
3387545993219,10cyclictest23-21ksoftirqd/108:45:031
338754599317,3cyclictest131rcu_preempt07:55:001
3387545993120,9cyclictest23-21ksoftirqd/107:45:001
3387545992915,12cyclictest23-21ksoftirqd/108:50:011
338755599282,5cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional