You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 03:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Apr 09, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1002699561,33cyclictest23329-21ssh21:25:122
10026995123,19cyclictest27-21ksoftirqd/221:10:122
1000299512,15cyclictest13348-21ssh00:00:110
1002699502,6cyclictest131rcu_preempt23:25:112
1002699489,28cyclictest0-21swapper/221:45:122
1001499463,14cyclictest26531-21ssh00:25:131
1000299441,32cyclictest23686-21ssh23:30:120
1001499431,33cyclictest18300-21ssh21:40:121
10037994124,9cyclictest0-21swapper/320:05:133
1003799402,26cyclictest28366irq/58-eth000:10:123
1003799393,21cyclictest28366irq/58-eth022:45:173
1003799393,21cyclictest28366irq/58-eth022:45:173
1003799384,19cyclictest28366irq/58-eth021:50:133
1003799383,24cyclictest28366irq/58-eth021:15:143
1002699365,22cyclictest0-21swapper/221:55:132
1003799351,21cyclictest28366irq/58-eth023:35:123
1002699355,10cyclictest131rcu_preempt20:50:132
1002699352,15cyclictest5088-21ssh21:15:142
1003799332,16cyclictest28366irq/58-eth021:20:133
10026993312,7cyclictest9991-21cyclictest20:10:132
1002699323,12cyclictest17114-21ssh22:15:142
1002699321,21cyclictest19905-21ssh23:10:122
1003799313,22cyclictest0-21swapper/322:25:153
1003799311,17cyclictest0-21swapper/321:45:133
1002699317,16cyclictest22346irq/40-dwc2_hso22:20:112
10026993112,13cyclictest924-21snmpd22:30:142
1002699311,19cyclictest924-21snmpd21:30:122
1001499314,15cyclictest1839-21ssh23:00:141
1001499311,23cyclictest21103-21sh22:35:131
1000299313,20cyclictest27531-21ssh23:50:130
1002699303,16cyclictest261rcuc/222:50:122
1001499302,20cyclictest23677-21ssh23:30:111
10037992913,10cyclictest28366irq/58-eth021:10:123
1002699294,14cyclictest22346irq/40-dwc2_hso19:20:122
1002699292,20cyclictest0-21swapper/220:30:122
10037992814,9cyclictest28366irq/58-eth000:15:123
1002699283,15cyclictest22346irq/40-dwc2_hso20:40:122
1003799271,18cyclictest0-21swapper/322:05:123
1002699273,17cyclictest22346irq/40-dwc2_hso20:05:122
1002699273,15cyclictest22346irq/40-dwc2_hso20:15:122
1002699273,14cyclictest22346irq/40-dwc2_hso23:15:112
1002699273,14cyclictest22346irq/40-dwc2_hso21:00:122
1002699271,21cyclictest20530-21ssh00:39:392
1002699271,21cyclictest20530-21ssh00:39:392
1002699271,12cyclictest261rcuc/223:05:122
10026992710,14cyclictest28490-21sshd22:38:302
1000299272,20cyclictest30225-21ssh22:40:130
1003799265,12cyclictest28366irq/58-eth023:25:123
10037992614,8cyclictest28366irq/58-eth021:25:123
10026992617,5cyclictest29170-21ssh23:50:192
10014992610,11cyclictest9991-21cyclictest20:20:121
10002992614,8cyclictest23433-21df21:25:130
1003799253,14cyclictest28466irq/59-eth022:40:113
1002699252,16cyclictest8254-21sh00:15:122
1002699251,17cyclictest15234-21irqcore19:15:162
10026992510,7cyclictest27977-21ls21:10:002
10026992510,7cyclictest27977-21ls21:10:002
1001499251,7cyclictest9150-21apt21:35:101
1001499251,20cyclictest20939-21ssh22:50:321
1000299253,14cyclictest0-21swapper/020:00:130
1003799241,15cyclictest28366irq/58-eth023:20:133
1003799241,13cyclictest28366irq/58-eth019:10:123
1002699242,17cyclictest22346irq/40-dwc2_hso22:25:152
10026992416,5cyclictest25198-21ssh22:00:212
10026992411,9cyclictest27-21ksoftirqd/222:55:202
10026992411,9cyclictest0-21swapper/220:25:192
1002699241,13cyclictest1810-21/usr/sbin/munin23:00:222
1002699241,10cyclictest1009466ssh23:20:312
1001499241,17cyclictest32735-21ssh23:35:131
1000299248,2cyclictest131rcu_preempt21:35:020
1000299242,15cyclictest9431-21ssh23:40:120
1000299241,10cyclictest0-21swapper/023:21:320
1003799232,15cyclictest28366irq/58-eth000:05:133
1002699232,12cyclictest11441-21kthreadcore23:40:192
1002699231,5cyclictest0-21swapper/200:05:152
1002699231,17cyclictest22061-21gpgv00:05:012
1002699231,14cyclictest22346irq/40-dwc2_hso00:10:122
1002699231,13cyclictest9180-21rm21:35:112
1002699231,12cyclictest14258-21sh21:20:142
1002699231,11cyclictest924-21snmpd22:05:122
1001499231,16cyclictest6974-21ssh22:45:191
1001499231,16cyclictest6974-21ssh22:45:181
1000299233,15cyclictest3156-21ssh00:30:150
10002992313,6cyclictest0-21swapper/022:10:130
10002992311,7cyclictest0-21swapper/022:35:130
10002992311,3cyclictest27976-21diskmemload00:13:390
1003799229,6cyclictest0-21swapper/320:40:123
1003799224,10cyclictest0-21swapper/321:40:123
1002699223,4cyclictest131rcu_preempt22:40:122
1002699222,12cyclictest4272-21ssh23:55:122
1002699221,9cyclictest11103-21tail22:45:342
1002699221,9cyclictest11103-21tail22:45:332
10026992212,4cyclictest0-21swapper/219:45:232
1002699221,11cyclictest29079-21munin-run19:30:012
1001499225,14cyclictest18494-21ssh23:45:121
10014992215,5cyclictest21-21ksoftirqd/100:35:121
10014992215,5cyclictest21-21ksoftirqd/100:35:121
1000299221,18cyclictest15027-21uniq19:15:140
1003799213,11cyclictest0-21swapper/300:25:053
1002699217,9cyclictest27-21ksoftirqd/222:10:142
1002699214,14cyclictest29952-21kthreadcore20:35:222
1002699214,10cyclictest27-21ksoftirqd/220:55:122
1002699212,9cyclictest25447-21cron20:00:012
1002699212,9cyclictest25447-21cron20:00:002
1002699212,15cyclictest0-21swapper/221:50:142
1002699212,12cyclictest26202-21kthreadcore23:30:212
1002699211,9cyclictest8937-21cron23:40:012
1002699211,7cyclictest0-21swapper/220:25:012
1002699211,11cyclictest20878-21kthreadcore21:40:222
1002699211,10cyclictest2736-21munin-run00:30:012
1002699211,10cyclictest18511-21ssh23:45:132
10026992110,9cyclictest23213-21sshd00:21:392
1001499211,15cyclictest3105-21cut22:25:171
1001499211,15cyclictest13977-21cut20:55:161
1000299213,9cyclictest131rcu_preempt23:55:190
1000299211,6cyclictest15805-21ntp_192.168.11400:35:250
1000299211,6cyclictest15805-21ntp_192.168.11400:35:240
10002992111,4cyclictest27976-21diskmemload00:19:230
10002992111,3cyclictest27976-21diskmemload21:45:030
1000299210,15cyclictest19977-21ls19:20:170
1003799209,6cyclictest0-21swapper/323:55:343
1003799208,7cyclictest28366irq/58-eth022:35:143
1003799206,9cyclictest28366irq/58-eth023:10:133
10037992014,4cyclictest0-21swapper/300:00:213
10037992011,6cyclictest0-21swapper/321:30:493
1003799201,11cyclictest0-21swapper/300:35:123
1003799201,11cyclictest0-21swapper/300:35:123
1002699207,5cyclictest13077-21kthreadcore19:10:202
1002699207,10cyclictest6211-21cron19:40:012
1002699202,9cyclictest6667-21fschecks_time19:40:152
1002699201,9cyclictest5366-21kthreadcore20:45:182
1002699201,9cyclictest29625-21grep19:30:172
1002699201,9cyclictest29625-21grep19:30:162
1002699201,15cyclictest6809-21ntp_kernel_pll_00:30:262
1002699201,15cyclictest27093-21kthreadcore20:00:192
1002699200,14cyclictest16221-21cut19:50:142
10014992012,5cyclictest13203-21ssh21:55:131
1000299207,3cyclictest27976-21diskmemload22:54:490
1000299206,3cyclictest131rcu_preempt21:35:200
1000299202,10cyclictest0-21swapper/019:25:120
1000299201,7cyclictest23157-21grep23:10:210
1000299201,13cyclictest24191-21missed_timers19:55:220
1000299201,13cyclictest24191-21missed_timers19:55:220
10002992010,3cyclictest27976-21diskmemload23:49:060
10002992010,3cyclictest27976-21diskmemload23:06:210
10002992010,3cyclictest27976-21diskmemload21:16:280
1000299200,17cyclictest11054-21missed_timers20:15:220
1003799198,9cyclictest0-21swapper/322:34:173
1003799193,10cyclictest0-21swapper/322:00:133
10037991911,6cyclictest0-21swapper/322:52:053
1003799191,13cyclictest0-21swapper/320:50:163
1003799191,13cyclictest0-21swapper/300:30:243
1003799191,12cyclictest0-21swapper/320:25:283
1003799191,11cyclictest0-21swapper/320:00:263
1003799191,10cyclictest0-21swapper/321:35:103
1003799191,10cyclictest0-21swapper/319:20:113
10037991910,7cyclictest0-21swapper/300:20:283
1001499199,7cyclictest29304-21unixbench_multi22:55:371
1001499198,9cyclictest11814-21cron00:35:011
10014991912,5cyclictest15515-21ssh00:00:211
10014991911,6cyclictest6501-21cron22:45:011
10014991911,5cyclictest6212-21cron19:40:001
10014991911,5cyclictest13581-21cron20:55:011
10014991910,6cyclictest0-21swapper/100:15:221
1000299198,8cyclictest31374-21cut00:10:020
1000299198,2cyclictest12-21ksoftirqd/022:15:220
1000299194,9cyclictest0-21swapper/019:50:110
1000299194,4cyclictest12-21ksoftirqd/020:50:300
1000299192,13cyclictest14614-21ssh23:25:130
1000299191,4cyclictest5183-21munin-run23:20:010
1000299191,15cyclictest13976-21grep20:55:170
1000299191,14cyclictest12153-21ssh22:30:160
1000299191,10cyclictest27976-21diskmemload22:55:040
10002991910,6cyclictest0-21swapper/022:25:150
10002991910,3cyclictest27976-21diskmemload00:23:040
1003799189,6cyclictest0-21swapper/323:06:573
1003799188,6cyclictest0-21swapper/322:10:133
1003799184,11cyclictest0-21swapper/323:20:033
1003799182,10cyclictest0-21swapper/323:40:053
1003799182,10cyclictest0-21swapper/322:15:133
1003799182,10cyclictest0-21swapper/321:00:133
1003799182,10cyclictest0-21swapper/320:45:043
1003799182,10cyclictest0-21swapper/320:20:133
1003799182,10cyclictest0-21swapper/320:15:143
1003799182,10cyclictest0-21swapper/319:35:143
10037991812,4cyclictest0-21swapper/323:30:113
10037991812,4cyclictest0-21swapper/322:20:233
10037991810,6cyclictest0-21swapper/323:53:243
10037991810,3cyclictest0-21swapper/322:58:463
1002699182,8cyclictest9991-21cyclictest19:10:012
1002699182,8cyclictest9991-21cyclictest19:10:012
1001499189,6cyclictest21-21ksoftirqd/122:00:131
1001499189,6cyclictest14230-21apt-get21:20:131
1001499188,7cyclictest18295-21latency_hist23:45:021
1001499186,10cyclictest5041-21sh21:50:191
1001499183,13cyclictest12129-21kthreadcore23:05:191
1001499182,10cyclictest0-21swapper/119:50:111
1001499181,9cyclictest20856-21/usr/sbin/munin19:55:291
1001499181,9cyclictest20856-21/usr/sbin/munin19:55:291
1001499181,4cyclictest0-21swapper/119:45:021
1001499181,14cyclictest17377-21apt-get20:25:131
1001499181,13cyclictest5599-21ssh23:20:131
1001499181,12cyclictest27530-21sh23:50:141
10014991810,5cyclictest5774-21ssh21:15:201
1001499180,9cyclictest15379-21kthreadcore23:25:201
1001499180,10cyclictest25145-21sed19:25:191
1000299189,7cyclictest2382-21cut23:35:210
1000299182,4cyclictest12-21ksoftirqd/000:28:450
1000299182,11cyclictest0-21swapper/020:35:140
1000299181,9cyclictest18600-21/usr/sbin/munin21:00:160
10002991812,4cyclictest26693-21irqcore22:20:160
10002991811,5cyclictest0-21swapper/022:05:180
1000299181,13cyclictest4157-21ssh21:50:130
1000299181,12cyclictest14337-21ssh21:20:140
1003799179,6cyclictest28366irq/58-eth021:55:143
1003799172,9cyclictest0-21swapper/320:55:153
1003799172,9cyclictest0-21swapper/319:45:133
1003799172,10cyclictest0-21swapper/323:00:163
1003799172,10cyclictest0-21swapper/319:55:143
1003799172,10cyclictest0-21swapper/319:55:133
1003799172,10cyclictest0-21swapper/319:40:133
10037991711,4cyclictest0-21swapper/323:46:113
1001499179,6cyclictest7669-21cron22:10:031
1001499179,5cyclictest27980-21cron21:10:001
1001499179,5cyclictest27980-21cron21:10:001
1001499176,9cyclictest28162-21kthreadcore22:20:191
1001499176,8cyclictest31558-21apt-get00:10:121
1001499174,7cyclictest0-21swapper/119:30:101
1001499174,7cyclictest0-21swapper/119:30:101
1001499172,4cyclictest4866-21users21:30:301
1001499172,12cyclictest23510-21sh23:10:231
1001499172,11cyclictest4282-21ssh23:55:131
1001499172,10cyclictest26158-21ssh00:05:251
1001499171,8cyclictest0-21swapper/122:16:171
10014991712,3cyclictest14662-21kworker/1:119:15:111
10014991711,4cyclictest3553-21usage_EthcTask20:40:301
10014991711,4cyclictest31717-21interrupts22:05:161
10014991711,4cyclictest19343-21kthreadcore00:20:211
1001499171,12cyclictest28637-21/usr/sbin/munin21:10:151
1001499171,11cyclictest30950-21cut23:15:191
1001499171,11cyclictest13028-21ssh22:30:201
10014991710,5cyclictest8765-21cron20:50:011
1000299179,6cyclictest6570-21awk19:40:140
1000299178,6cyclictest1136-21tr20:05:210
1000299176,9cyclictest32271-21grep20:40:180
1000299171,7cyclictest30207-21ssh21:10:200
1000299171,5cyclictest53146usb-storage20:45:110
1000299171,3cyclictest0-21swapper/023:02:500
10002991711,4cyclictest27125-21vmstat22:00:310
10002991711,4cyclictest0-21swapper/020:10:300
1003799163,8cyclictest0-21swapper/320:35:133
1001499169,4cyclictest24075-21kthreadcore21:25:181
1001499168,6cyclictest28995-21sed20:00:211
1001499167,6cyclictest7903-21wc20:15:161
1001499166,8cyclictest11404-21tune2fs19:45:151
1001499163,11cyclictest5158-21kthreadcore20:10:201
1001499162,11cyclictest22160-21apt-get20:30:111
1001499161,12cyclictest30988-21ssh21:45:231
1001499161,12cyclictest10375-21wc19:10:151
1001499161,11cyclictest29947-21tr20:35:211
1001499161,11cyclictest19927-21fschecks_time19:20:151
10014991610,4cyclictest0-21swapper/121:00:231
1001499160,10cyclictest30563-21cpuspeed20:05:141
1000299163,10cyclictest19601-21kthreadcore21:40:190
1000299162,9cyclictest0-21swapper/019:10:130
1000299162,10cyclictest0-21swapper/020:20:130
1000299161,12cyclictest6969-21ssh22:45:180
1000299161,12cyclictest6969-21ssh22:45:180
1000299161,11cyclictest16791-21ssh21:55:250
10002991610,4cyclictest24080-21uname21:05:180
10002991610,4cyclictest24080-21uname21:05:170
1001499151,10cyclictest10068-21ls19:10:011
1001499151,10cyclictest10068-21ls19:10:011
1000299159,4cyclictest17492-21fschecks_count20:25:170
1000299152,11cyclictest13924-21kthreadcore19:45:200
1000299151,11cyclictest24144-21kthreadcore20:30:190
1000299151,10cyclictest30556-21kthreadcore19:30:180
1000299151,10cyclictest30556-21kthreadcore19:30:180
1003799149,3cyclictest0-21swapper/321:06:063
1003799149,3cyclictest0-21swapper/321:06:053
1003799141,7cyclictest0-21swapper/319:50:263
1000299142,9cyclictest10099-21latency_hist19:10:020
1000299142,9cyclictest10099-21latency_hist19:10:010
1000299141,11cyclictest1743-21diskstats19:35:150
1003799131,9cyclictest0-21swapper/320:30:183
1003799131,8cyclictest0-21swapper/319:25:123
1003799131,5cyclictest0-21swapper/320:10:173
1003799121,9cyclictest0-21swapper/319:30:203
1003799121,9cyclictest0-21swapper/319:30:203
1003799121,4cyclictest0-21swapper/319:15:283
1003799101,6cyclictest0-21swapper/319:10:023
1003799101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional