You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 30, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2521899902,48cyclictest70271-21munin-run22:40:010
25232998931,23cyclictest86801-21sh23:18:063
25223998728,32cyclictest3467602-21systemd-journal20:15:031
25218998528,13cyclictest98693-21ssh23:45:230
2521899852,40cyclictest65391-21ssh22:28:400
25218998427,15cyclictest97673-21ssh23:43:220
25218998427,15cyclictest97673-21ssh23:43:220
25232998315,35cyclictest59079-21sshd22:14:063
25218998125,52cyclictest12-21ksoftirqd/023:15:020
25218997934,41cyclictest16850irq/134-800000000:00:010
25218997934,41cyclictest16850irq/134-800000000:00:000
2521899793,12cyclictest68291-21grep22:35:020
2521899791,37cyclictest33307-21ssh21:14:260
25218997549,15cyclictest30854-21modprobe21:00:000
2521899742,10cyclictest59539-21cut22:15:010
25218997415,14cyclictest31385-21modprobe21:10:000
25218997414,17cyclictest102626-21ssh23:55:000
2521899731,38cyclictest44757-21ssh21:40:210
25218997234,7cyclictest44372-21sh21:40:010
25218997225,15cyclictest107881-21ssh00:06:490
25232997111,36cyclictest31397-21diskmemload23:22:563
25218997126,38cyclictest0-21swapper/000:25:000
2521899703,8cyclictest113470-21munin-run00:20:000
25218996926,13cyclictest95605-21ssh23:38:330
25232996828,35cyclictest64440-21sh22:26:143
25223996827,37cyclictest23-21ksoftirqd/122:10:431
25218996824,10cyclictest39880-21ssh21:29:440
25223996727,35cyclictest48836-21latency_hist21:50:021
25223996727,35cyclictest48836-21latency_hist21:50:021
25218996725,11cyclictest72760-21ssh22:45:190
25232996628,33cyclictest42243-21latency_hist21:35:023
25223996625,36cyclictest118650-21sh00:31:321
25218996619,12cyclictest52537-21ssh21:58:550
25232996526,34cyclictest91138-21sh23:28:163
25223996528,32cyclictest37070-21sh21:22:571
2523299641,33cyclictest110260-21ssh00:12:243
25218996421,39cyclictest3467602-21systemd-journal22:05:000
2523299632,33cyclictest59873-21ssh22:15:353
25223996326,34cyclictest63772-21latency_hist22:25:011
2522399632,16cyclictest114897-21ssh00:23:051
2522399631,16cyclictest108217-21sh00:07:371
25218996334,4cyclictest16950irq/224-800000023:05:550
25232996238,19cyclictest3467602-21systemd-journal21:30:023
25232996221,36cyclictest275-21dbus-daemon21:20:013
25227996218,11cyclictest30-21ksoftirqd/222:20:012
2522399622,15cyclictest97931-21sh23:44:011
2522399622,15cyclictest97931-21sh23:44:001
2522399622,15cyclictest90813-21sh23:27:171
2522399622,14cyclictest94053-21munin-run23:35:001
25218996228,11cyclictest80752-21ssh23:04:090
25218996223,8cyclictest309-21in:imuxsock20:25:000
25218996223,8cyclictest309-21in:imuxsock20:25:000
25218996214,13cyclictest106526-21ssh00:03:470
25232996133,24cyclictest98911-21sh23:45:553
2522399612,17cyclictest40381-21ssh21:30:221
25223996121,35cyclictest32027-21sh21:11:141
25223996115,8cyclictest131rcu_preempt20:55:001
25223996115,8cyclictest131rcu_preempt20:55:001
25218996129,10cyclictest12-21ksoftirqd/022:07:310
25218996129,10cyclictest12-21ksoftirqd/022:07:310
25218996128,11cyclictest86865-21ssh23:18:160
25232996016,40cyclictest27012-21ls19:45:023
25232996016,40cyclictest27012-21ls19:45:023
2523299601,54cyclictest26799-21wc19:40:013
2523299601,53cyclictest37956-21latency_hist21:25:013
25223996028,27cyclictest85323-21sh23:14:551
2522399601,18cyclictest69747-21ssh22:38:411
2522399601,17cyclictest42996-21ssh21:36:261
2522399601,14cyclictest45483-21ssh21:42:161
2521899602,41cyclictest118030-21ssh00:30:020
25232995920,34cyclictest115707-21cut00:25:013
2523299591,54cyclictest120202-21sh00:35:043
2522399592,51cyclictest101710-21ssh23:52:421
2522399592,17cyclictest64500-21sh22:26:231
2522399592,17cyclictest49412-21ssh21:51:231
2522399592,15cyclictest88573-21sh23:22:031
2522399591,17cyclictest77792-21ssh22:57:091
25218995948,7cyclictest16850irq/134-800000000:40:020
25218995927,12cyclictest37640-21ssh21:24:270
2523299582,51cyclictest32648-21ssh21:12:443
2523299582,50cyclictest74895-21ssh22:50:203
2523299581,52cyclictest72507-21latency_hist22:45:013
25227995847,7cyclictest25966-21kworker/2:319:35:022
25223995827,28cyclictest23-21ksoftirqd/122:19:221
2522399582,12cyclictest33923-21ssh21:15:291
25223995820,34cyclictest4146980-1kworker/u9:121:27:241
25223995817,37cyclictest51865-21rm21:57:091
25223995817,36cyclictest275-21dbus-daemon00:30:011
2522399581,6cyclictest102980-21ssh23:55:211
2522399581,6cyclictest102980-21ssh23:55:211
25223995815,39cyclictest27832-21cat20:00:021
25223995815,39cyclictest27832-21cat20:00:021
2522399581,13cyclictest48-21kcompactd000:00:231
25218995849,5cyclictest16950irq/224-800000023:25:000
25218995842,11cyclictest1-21systemd20:05:000
25218995826,13cyclictest71865-21ssh22:43:370
2523299572,50cyclictest70210-21cron22:40:003
2523299572,49cyclictest29831-21latency_hist20:40:013
2523299572,48cyclictest30106-21munin-plugin-st20:45:003
25232995724,29cyclictest30355-21latency_hist20:50:023
25232995722,31cyclictest37-21ksoftirqd/323:31:323
2523299571,51cyclictest94567-21ssh23:35:543
25223995725,28cyclictest86379-21sshd23:17:051
2522399572,50cyclictest100524-21munin-run23:50:011
25223995722,31cyclictest31397-21diskmemload22:03:221
2522399572,18cyclictest56414-21ssh22:07:481
2522399572,18cyclictest56414-21ssh22:07:481
2522399572,17cyclictest94345-21ssh23:35:101
2522399571,9cyclictest111880-21ssh00:16:031
2522399571,17cyclictest120742-21ssh00:36:291
2523299563,47cyclictest28796-21munin-run20:20:003
2523299562,49cyclictest42468-21sshd21:35:053
2523299562,49cyclictest31230-21latency_hist21:05:023
2523299562,49cyclictest27678-21latency_hist19:55:023
2523299562,26cyclictest63090-21ssh22:23:133
25232995622,30cyclictest37-21ksoftirqd/300:34:303
2523299561,50cyclictest102244-21sh23:53:593
25227995644,8cyclictest30-21ksoftirqd/200:20:012
2522399566,46cyclictest110644-21ssh00:13:191
2522399562,48cyclictest74768-21grep22:50:021
25218995650,4cyclictest0-21swapper/021:33:070
25218995627,13cyclictest111146-21ssh00:14:400
25218995625,14cyclictest76077-21ssh22:53:190
2521899561,4cyclictest131rcu_preempt21:15:400
25232995527,24cyclictest37-21ksoftirqd/321:57:113
2523299552,48cyclictest80145-21rm23:02:323
2523299552,48cyclictest78425-21sh22:58:473
2523299552,48cyclictest25751-21date19:20:013
2523299552,47cyclictest30850-21cron21:00:003
25232995521,31cyclictest66361-21kworker/u8:022:34:233
2523299551,50cyclictest46381-21ssh21:44:333
2523299551,49cyclictest81980-21ssh23:06:453
2522799555,11cyclictest30-21ksoftirqd/223:44:532
2522799555,11cyclictest30-21ksoftirqd/223:44:532
2522399552,16cyclictest66252-21ssh22:30:211
25218995528,12cyclictest91130-21ssh23:28:150
25218995526,8cyclictest27252-21unin-run19:50:000
25218995523,28cyclictest12-21ksoftirqd/021:46:020
25218995523,28cyclictest12-21ksoftirqd/021:46:020
2523299542,48cyclictest98479-21cut23:45:033
2523299542,48cyclictest98479-21cut23:45:023
2523299542,48cyclictest108522-21sh00:08:263
2523299542,46cyclictest50079-21ssh21:53:093
25232995419,31cyclictest3467602-21systemd-journal21:07:053
2523299541,48cyclictest0-21swapper/321:45:453
2523299541,48cyclictest0-21swapper/321:45:453
2523299541,47cyclictest84507-21ssh23:12:433
2522799544,11cyclictest30-21ksoftirqd/223:48:112
2522399543,5cyclictest364-21snmpd19:46:331
2522399542,17cyclictest28526-21munin-run20:15:011
2522399541,9cyclictest76706-21ssh22:54:571
2522399541,8cyclictest79259-21ssh23:00:201
2522399541,8cyclictest71152-21ssh22:41:451
2522399541,4cyclictest82245-21ssh23:07:341
2522399541,15cyclictest27001-21unin-run19:45:011
2522399541,15cyclictest27001-21unin-run19:45:011
25218995435,16cyclictest16850irq/134-800000022:18:400
25218995425,14cyclictest119137-21ssh00:32:540
25232995327,23cyclictest37-21ksoftirqd/320:05:013
2523299532,46cyclictest73714-21ssh22:47:463
25232995318,31cyclictest3467602-21systemd-journal20:20:023
25232995318,31cyclictest3467602-21systemd-journal20:20:023
2523299531,48cyclictest112559-21sh00:17:443
2523299531,48cyclictest0-21swapper/319:25:013
2523299531,46cyclictest53455-21ssh22:00:403
25227995336,13cyclictest1-21systemd22:45:002
2522399534,44cyclictest30848-21cron21:00:001
2522399533,14cyclictest29573-21munin-run20:35:001
25223995311,37cyclictest0-21swapper/120:27:491
25218995349,2cyclictest0-21swapper/020:05:020
25232995227,21cyclictest55733-21sh22:06:023
25232995227,21cyclictest55733-21sh22:06:023
2523299522,44cyclictest25474-21wc19:15:023
2523299521,47cyclictest150-21pr/ttyS223:57:153
2523299521,47cyclictest150-21pr/ttyS223:57:153
2523299521,47cyclictest150-21pr/ttyS220:10:013
2523299521,47cyclictest107022-21latency_hist00:05:023
2522799527,9cyclictest30-21ksoftirqd/222:48:372
2522799527,8cyclictest30-21ksoftirqd/200:30:012
25227995244,5cyclictest18385-21kworker/2:023:06:002
25227995241,7cyclictest30-21ksoftirqd/223:55:002
2522799524,15cyclictest30-21ksoftirqd/221:00:002
25227995238,8cyclictest30-21ksoftirqd/221:50:342
25223995225,10cyclictest3467602-21systemd-journal19:15:001
25223995224,23cyclictest23-21ksoftirqd/119:55:011
2522399522,16cyclictest28338-21latency_hist20:10:021
2522399522,12cyclictest30104-21munin-run20:45:001
2522399521,16cyclictest31419-21unin-run21:10:011
2523299511,24cyclictest31397-21diskmemload00:26:463
25227995143,5cyclictest30900-21kworker/2:123:33:372
25227995143,5cyclictest30900-21kworker/2:122:12:392
25227995142,5cyclictest18385-21kworker/2:023:00:152
2522799514,12cyclictest30-21ksoftirqd/221:26:132
25227995140,7cyclictest291rcuc/221:43:252
25227995137,9cyclictest30-21ksoftirqd/220:50:002
2522799513,13cyclictest30-21ksoftirqd/221:25:002
25227995113,7cyclictest30-21ksoftirqd/221:17:382
2522399518,38cyclictest26888-21cat19:40:021
2522399512,42cyclictest29944-21latency_hist20:40:021
2522399512,16cyclictest364-21snmpd20:01:371
25218995133,14cyclictest275-21dbus-daemon22:25:020
25218995123,13cyclictest77725-21ssh22:57:000
2523299502,43cyclictest27757-21readlink20:00:003
2523299502,43cyclictest27757-21readlink20:00:003
2522799506,10cyclictest30-21ksoftirqd/222:08:392
2522799506,10cyclictest30-21ksoftirqd/222:08:392
2522799506,10cyclictest30-21ksoftirqd/200:22:042
2522799505,9cyclictest30-21ksoftirqd/200:11:072
2522799505,7cyclictest30-21ksoftirqd/223:57:282
2522799505,7cyclictest30-21ksoftirqd/223:57:282
2522799505,11cyclictest30-21ksoftirqd/223:20:002
25227995042,5cyclictest30900-21kworker/2:123:29:542
25227995042,5cyclictest102636-21kworker/2:300:34:022
25227995041,6cyclictest18385-21kworker/2:020:02:042
25223995019,27cyclictest23-21ksoftirqd/119:25:011
25218995046,2cyclictest0-21swapper/019:10:020
2523299491,44cyclictest29307-21cron20:30:003
2522799497,11cyclictest30-21ksoftirqd/222:54:332
2522799497,10cyclictest30-21ksoftirqd/222:22:162
2522799495,8cyclictest30-21ksoftirqd/221:33:062
25227994941,5cyclictest30900-21kworker/2:121:04:052
25227994941,5cyclictest28301-21kworker/2:220:22:152
25227994941,5cyclictest28301-21kworker/2:220:22:152
25227994941,5cyclictest25966-21kworker/2:319:28:362
25227994941,5cyclictest18385-21kworker/2:023:35:272
25227994941,5cyclictest18385-21kworker/2:022:27:232
25227994941,5cyclictest18385-21kworker/2:020:35:312
25227994941,5cyclictest18385-21kworker/2:019:59:272
25227994941,5cyclictest18385-21kworker/2:019:59:272
25227994940,5cyclictest18385-21kworker/2:021:57:032
2522799493,9cyclictest30-21ksoftirqd/200:03:292
2522799491,12cyclictest30-21ksoftirqd/221:14:052
25227994910,10cyclictest30-21ksoftirqd/221:50:022
25227994910,10cyclictest30-21ksoftirqd/221:50:022
2522399493,41cyclictest0-21swapper/119:25:051
2522399492,7cyclictest29208-21wc20:25:021
2522399492,7cyclictest29208-21wc20:25:021
25218994931,13cyclictest12-21ksoftirqd/020:55:000
25218994931,13cyclictest12-21ksoftirqd/020:55:000
2523299482,42cyclictest25317-21sed19:10:023
25232994823,22cyclictest37-21ksoftirqd/320:15:013
2523299481,42cyclictest26658-21latency_hist19:35:023
2523299481,42cyclictest26346-21latency_hist19:30:023
2522799487,9cyclictest30-21ksoftirqd/222:03:362
25227994841,5cyclictest25966-21kworker/2:319:35:032
25227994840,5cyclictest30900-21kworker/2:122:37:252
25227994840,5cyclictest30900-21kworker/2:122:31:322
25227994840,5cyclictest30900-21kworker/2:121:36:062
25227994840,5cyclictest28301-21kworker/2:220:25:092
25227994840,5cyclictest25966-21kworker/2:320:10:052
25227994840,5cyclictest18385-21kworker/2:023:12:072
25227994840,5cyclictest18385-21kworker/2:022:57:282
25227994840,5cyclictest18385-21kworker/2:020:51:412
25227994840,5cyclictest18385-21kworker/2:020:51:412
25227994840,5cyclictest18385-21kworker/2:020:44:192
25227994840,5cyclictest18385-21kworker/2:019:53:242
25227994840,5cyclictest102636-21kworker/2:300:37:492
25227994839,5cyclictest28301-21kworker/2:220:15:482
2522799483,7cyclictest30-21ksoftirqd/200:06:302
2522799482,8cyclictest30-21ksoftirqd/223:21:222
2522399483,39cyclictest0-21swapper/119:20:001
2522399482,9cyclictest1-21systemd20:50:011
25218994833,10cyclictest12-21ksoftirqd/019:25:000
2521899481,7cyclictest94257-21latency_hist23:35:030
25218994813,30cyclictest29572-21modprobe20:35:000
2523299472,39cyclictest27381-21latency_hist19:50:013
2523299471,42cyclictest150-21pr/ttyS220:35:003
25227994740,4cyclictest18385-21kworker/2:019:15:172
25227994739,5cyclictest18385-21kworker/2:019:45:062
25227994739,5cyclictest18385-21kworker/2:019:42:062
25227994739,5cyclictest18385-21kworker/2:019:42:062
25227994739,5cyclictest18385-21kworker/2:019:20:422
25227994739,5cyclictest18385-21kworker/2:019:10:382
25227994738,5cyclictest30900-21kworker/2:121:05:142
2522399472,4cyclictest0-21swapper/119:31:301
25218994739,4cyclictest16850irq/134-800000021:55:000
25218994727,13cyclictest275-21dbus-daemon20:00:000
25218994727,13cyclictest275-21dbus-daemon20:00:000
25232994617,24cyclictest30801-21latency_hist20:55:023
25232994617,24cyclictest30801-21latency_hist20:55:023
2522799466,10cyclictest30-21ksoftirqd/220:06:372
25227994638,5cyclictest28301-21kworker/2:220:30:412
2522399463,37cyclictest31128-21cron21:05:001
25218994624,17cyclictest1-21systemd19:35:000
2521899451,15cyclictest28676-21latency_hist20:15:020
25218994428,10cyclictest26719-21systemctl19:40:000
25218994422,16cyclictest275-21dbus-daemon20:30:000
2522399433,35cyclictest25225-21latency_hist19:10:011
25218994327,12cyclictest12-21ksoftirqd/020:15:030
25218994229,8cyclictest26192-21cron19:30:000
25218994027,10cyclictest12-21ksoftirqd/020:40:030
25218994024,12cyclictest12-21ksoftirqd/019:20:010
2521899395,20cyclictest27493-21munin-plugin-st19:55:000
25218993928,6cyclictest16950irq/224-800000020:50:010
25218993832,4cyclictest16950irq/224-800000019:15:010
2521899381,6cyclictest12-21ksoftirqd/021:05:000
25218993628,5cyclictest19350irq/34-mmc120:40:310
25218993523,8cyclictest3467602-21systemd-journal19:44:180
25218993523,8cyclictest3467602-21systemd-journal19:44:180
25227993222,7cyclictest30-21ksoftirqd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional