You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-17 - 00:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Wed Jul 16, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
105102980,2sleep20-21swapper/209:21:592
2230599907,32cyclictest13030-21ssh10:50:193
56162860,2sleep00-21swapper/010:40:280
2230299862,70cyclictest8763-21cpuspeed09:20:240
219732860,2sleep00-21swapper/009:38:120
2230299842,56cyclictest8585-21ssh11:25:060
2230299842,56cyclictest8585-21ssh11:25:060
23722800,2sleep00-21swapper/011:16:110
22302998022,39cyclictest9374-21modprobe10:05:050
229942770,5sleep12230399cyclictest11:01:471
229942770,5sleep12230399cyclictest11:01:471
2230299762,57cyclictest0-21swapper/007:25:050
22302997615,47cyclictest30188-21sed12:30:340
255782750,2sleep10-21swapper/110:25:231
2230299742,60cyclictest0-21swapper/010:21:200
22302997319,2cyclictest0-21swapper/009:00:290
22302997314,3cyclictest7950irq/56-dwc_otg_12:05:210
22302997230,5cyclictest8050irq/56-dwc_otg_10:55:170
2230299722,23cyclictest0-21swapper/011:04:090
2230299722,23cyclictest0-21swapper/011:04:090
2230299721,55cyclictest0-21swapper/009:42:550
2230299721,37cyclictest0-21swapper/008:15:200
2230299721,37cyclictest0-21swapper/008:15:200
2230299711,57cyclictest9826-21processes10:45:310
51812700,2sleep35180-21diskstats10:40:233
22302997010,49cyclictest0-21swapper/008:40:120
2230299691,20cyclictest26895-21sh12:26:490
22302996911,45cyclictest0-21swapper/007:32:270
22302996910,4cyclictest7850irq/56-dwc_otg09:45:340
22302996910,4cyclictest7850irq/56-dwc_otg09:45:340
2230299687,13cyclictest0-21swapper/010:27:550
2230299682,51cyclictest0-21swapper/009:51:120
2230299681,19cyclictest0-21swapper/011:25:250
2230299681,13cyclictest0-21swapper/010:37:400
2230299673,17cyclictest0-21swapper/011:45:310
22302996720,33cyclictest7850irq/56-dwc_otg10:15:340
22302996713,11cyclictest32612-21systemd-journal09:59:330
2230299671,11cyclictest8050irq/56-dwc_otg_12:35:110
2230299671,11cyclictest8050irq/56-dwc_otg_12:35:110
22302996711,10cyclictest17074-21apt-get08:25:200
2230299668,55cyclictest0-21swapper/012:00:200
22302996620,35cyclictest0-21swapper/011:30:180
2230299656,2cyclictest0-21swapper/010:35:050
2230299656,2cyclictest0-21swapper/010:35:050
22302996518,38cyclictest0-21swapper/012:21:280
2230299651,12cyclictest0-21swapper/011:57:380
2230299649,2cyclictest0-21swapper/007:10:320
2230299641,13cyclictest1139-21/usr/sbin/munin09:10:210
2230299632,18cyclictest0-21swapper/010:10:350
22302996313,10cyclictest661-21runrttasks12:16:050
2230299631,12cyclictest0-21swapper/012:11:310
2230299631,12cyclictest0-21swapper/007:55:180
2230299631,12cyclictest0-21swapper/007:55:180
52112620,9sleep2301ktimersoftd/209:15:232
2230299629,9cyclictest22753-21rm11:41:200
2230299629,9cyclictest22753-21rm11:41:200
2230299628,9cyclictest0-21swapper/009:18:190
2230299622,6cyclictest5471-21grep07:50:340
2230299621,4cyclictest0-21swapper/011:35:210
22302996212,28cyclictest17056-21unixbench_singl09:30:380
2230299621,20cyclictest10013-21fschecks_count10:05:250
2230299621,17cyclictest0-21swapper/010:53:370
2230299612,18cyclictest0-21swapper/009:26:050
2230299612,15cyclictest29119-21/usr/sbin/munin11:50:320
2230299592,4cyclictest0-21swapper/011:09:560
2230299592,17cyclictest0-21swapper/007:27:520
22302995910,10cyclictest689-21snmpd08:34:160
2230299581,35cyclictest25706-21munin-run07:20:050
22302995812,10cyclictest32722-21rm11:14:530
2230299581,19cyclictest0-21swapper/007:45:160
2230299581,15cyclictest0-21swapper/008:04:590
60862570,6sleep2301ktimersoftd/210:40:342
22302995713,9cyclictest0-21swapper/009:09:020
22302995711,4cyclictest0-21swapper/008:36:520
2230599565,18cyclictest8050irq/56-dwc_otg_12:15:213
22302995618,4cyclictest0-21swapper/008:55:120
22302995614,3cyclictest0-21swapper/008:45:450
22302995612,10cyclictest0-21swapper/007:42:040
2230299561,15cyclictest366-21chrt07:39:120
2230399553,12cyclictest7950irq/56-dwc_otg_07:50:171
22302995524,18cyclictest8050irq/56-dwc_otg_08:20:230
2230299551,14cyclictest0-21swapper/008:05:200
2230299551,13cyclictest0-21swapper/008:54:190
22302995412,10cyclictest0-21swapper/008:13:340
2230599538,7cyclictest7950irq/56-dwc_otg_08:20:183
2230599511,10cyclictest0-21swapper/308:00:193
2230599507,21cyclictest0-21swapper/310:05:223
22305995021,5cyclictest8050irq/56-dwc_otg_09:20:213
2230599499,20cyclictest7377-21ssh12:02:573
2230499497,6cyclictest0-21swapper/207:25:192
2230399497,5cyclictest7850irq/56-dwc_otg10:10:161
2230399494,23cyclictest0-21swapper/111:53:301
22305994813,15cyclictest8050irq/56-dwc_otg_10:35:053
22305994813,15cyclictest8050irq/56-dwc_otg_10:35:053
2230499486,11cyclictest0-21swapper/207:55:182
2230499486,11cyclictest0-21swapper/207:55:182
2230499483,6cyclictest0-21swapper/207:50:172
2230399477,18cyclictest7950irq/56-dwc_otg_11:30:211
106822470,5sleep22230499cyclictest11:26:272
2230599467,25cyclictest0-21swapper/312:30:203
2230599467,24cyclictest0-21swapper/309:25:213
2230499462,12cyclictest0-21swapper/208:35:182
218842463,12sleep00-21swapper/007:08:130
218842463,12sleep00-21swapper/007:08:130
2230599457,24cyclictest18695-21rm10:56:143
2230599457,24cyclictest0-21swapper/309:55:223
22305994512,20cyclictest17771-21ssh09:32:053
2230499452,6cyclictest0-21swapper/210:10:162
2230599447,24cyclictest0-21swapper/309:40:193
2230599446,24cyclictest0-21swapper/307:15:203
2230599445,11cyclictest7950irq/56-dwc_otg_11:50:193
22305994423,4cyclictest7950irq/56-dwc_otg_10:40:053
2230499441,3cyclictest7850irq/56-dwc_otg10:35:192
234082430,5sleep12230399cyclictest12:22:511
2230599438,18cyclictest5395-21ssh11:20:273
2230599438,18cyclictest5395-21ssh11:20:273
2230599437,23cyclictest0-21swapper/311:55:193
2230599436,21cyclictest9419-21ssh11:25:263
2230599435,20cyclictest18929-1kworker/u9:112:37:043
2230599435,20cyclictest18929-1kworker/u9:112:37:043
22305994322,8cyclictest8050irq/56-dwc_otg_10:20:213
2230599431,27cyclictest0-21swapper/308:55:123
2230499434,22cyclictest8050irq/56-dwc_otg_11:05:062
2230499434,22cyclictest8050irq/56-dwc_otg_11:05:062
2230399438,31cyclictest8050irq/56-dwc_otg_07:55:181
2230399438,31cyclictest8050irq/56-dwc_otg_07:55:181
22303994312,27cyclictest23-21ksoftirqd/109:00:171
224452420,5sleep12230399cyclictest07:10:191
2230599429,15cyclictest30727-21chrt07:30:343
2230599428,8cyclictest9766-21ntp_states10:45:303
2230599427,25cyclictest0-21swapper/308:25:203
2230599427,11cyclictest28710-21ssh11:09:563
2230599426,11cyclictest2417-21ssh09:10:423
2230599425,18cyclictest27060-21chrt12:27:083
22305994216,4cyclictest7850irq/56-dwc_otg07:44:073
22305994212,19cyclictest24138-21ssh11:43:573
22305994212,19cyclictest24138-21ssh11:43:573
22305994210,8cyclictest27950-21df07:25:223
22304994214,4cyclictest7950irq/56-dwc_otg_10:50:052
22303994218,5cyclictest29725-21tr12:30:291
2230599418,19cyclictest29751-21ssh09:48:493
2230599418,19cyclictest29751-21ssh09:48:493
2230599418,17cyclictest11598-21ssh12:08:203
2230599415,10cyclictest4460-21modprobe11:20:053
2230499418,24cyclictest0-21swapper/208:10:192
22304994123,5cyclictest25356-21ssh11:45:242
22304994118,4cyclictest21138-21aten_rbpower_po11:40:232
22304994118,4cyclictest21138-21aten_rbpower_po11:40:232
22304994117,8cyclictest0-21swapper/207:45:282
2230599409,22cyclictest7344-21sshd10:00:553
2230599409,13cyclictest14535-21/usr/sbin/munin10:10:323
2230599407,25cyclictest0-21swapper/307:10:183
22305994011,13cyclictest22969-21date09:40:063
22305994010,12cyclictest29806-21apt-get09:00:183
2230399407,24cyclictest0-21swapper/112:10:191
2230599397,5cyclictest8050irq/56-dwc_otg_11:30:063
2230599392,21cyclictest7950irq/56-dwc_otg_08:51:593
22305993911,12cyclictest5027-21ntp_states07:50:303
22305993910,12cyclictest17208-21/usr/sbin/munin11:35:213
2230499399,4cyclictest0-21swapper/210:25:322
2230499399,18cyclictest7950irq/56-dwc_otg_10:50:342
2230499391,25cyclictest19757-21ls08:30:342
2230399397,22cyclictest0-21swapper/107:25:201
2230599387,12cyclictest8050irq/56-dwc_otg_09:51:323
22305993818,7cyclictest8050irq/56-dwc_otg_11:04:403
22305993818,7cyclictest8050irq/56-dwc_otg_11:04:403
2230599381,26cyclictest7950irq/56-dwc_otg_12:21:473
2230599381,23cyclictest10738-21ls08:05:373
22305993812,13cyclictest7165-21ssh09:17:493
22305993810,12cyclictest27568-21latency_hist07:25:053
2230499387,23cyclictest0-21swapper/208:25:202
2230499386,8cyclictest7950irq/56-dwc_otg_12:40:062
2230499386,8cyclictest7950irq/56-dwc_otg_12:40:062
2230499383,8cyclictest1037-21fschecks_count11:55:232
2230499381,20cyclictest723-21diskmemload11:50:192
22303993817,6cyclictest2447-21sh11:56:191
2230399381,5cyclictest2022-21head11:15:351
2230599372,3cyclictest0-21swapper/308:40:183
22305993711,12cyclictest20227-21ssh10:18:533
22305993710,13cyclictest32664-21ssh11:14:463
22304993714,6cyclictest301ktimersoftd/209:30:352
2230499371,22cyclictest8258-21uniq08:00:232
22304993711,5cyclictest0-21swapper/211:25:052
22304993711,5cyclictest0-21swapper/211:25:052
2230399372,5cyclictest0-21swapper/108:10:181
2230399371,3cyclictest7850irq/56-dwc_otg10:40:191
22305993616,12cyclictest7950irq/56-dwc_otg_10:26:433
22305993610,15cyclictest723-21diskmemload11:49:133
2230499369,12cyclictest8050irq/56-dwc_otg_09:45:212
2230499369,12cyclictest8050irq/56-dwc_otg_09:45:212
2230499363,6cyclictest8050irq/56-dwc_otg_10:33:552
2230499363,6cyclictest8050irq/56-dwc_otg_10:33:552
2230499363,6cyclictest7950irq/56-dwc_otg_11:15:062
22304993615,5cyclictest661-21runrttasks11:18:102
2230499361,21cyclictest31776-21cpuspeed09:05:212
2230399364,16cyclictest22989-21grep09:40:061
2230399363,3cyclictest7950irq/56-dwc_otg_10:50:201
2230399363,12cyclictest7850irq/56-dwc_otg09:45:191
2230399363,12cyclictest7850irq/56-dwc_otg09:45:191
2230399362,23cyclictest0-21swapper/111:05:351
22303993613,6cyclictest32612-21systemd-journal07:39:061
2230599358,7cyclictest8050irq/56-dwc_otg_07:45:233
22305993512,13cyclictest8050irq/56-dwc_otg_12:10:403
22304993516,5cyclictest25656-21sh09:43:042
2230399359,4cyclictest0-21swapper/110:30:221
2230399359,4cyclictest0-21swapper/110:30:221
2230399352,20cyclictest0-21swapper/107:21:411
2230399352,13cyclictest7850irq/56-dwc_otg10:00:351
2230399351,25cyclictest5346-21latency11:20:261
2230399351,25cyclictest5346-21latency11:20:261
22303993511,10cyclictest5794-21timerandwakeup09:15:311
2230599342,20cyclictest0-21swapper/308:15:093
2230599342,20cyclictest0-21swapper/308:15:093
2230599341,25cyclictest32202-21missed_timers09:05:273
2230499348,14cyclictest8050irq/56-dwc_otg_08:15:292
2230499348,14cyclictest8050irq/56-dwc_otg_08:15:292
22304993423,8cyclictest22490-21grep10:20:372
22304993421,5cyclictest22906-21sh09:40:052
22304993418,6cyclictest689-21snmpd08:49:182
22304993417,4cyclictest27508-21sh07:25:052
2230499341,5cyclictest2520050ssh12:25:232
22304993415,9cyclictest661-21runrttasks08:42:222
22304993414,5cyclictest20613-21sh10:19:412
22304993413,7cyclictest0-21swapper/212:30:322
2230499341,15cyclictest12806-21irqstats09:25:282
22303993417,5cyclictest31677-21sh11:12:551
2230399341,4cyclictest8050irq/56-dwc_otg_12:05:191
22303993412,12cyclictest0-21swapper/110:22:401
186672340,2sleep10-21swapper/111:36:171
2230599336,5cyclictest7950irq/56-dwc_otg_08:35:353
2230599335,4cyclictest8050irq/56-dwc_otg_08:49:033
2230599331,22cyclictest11909-21cut08:10:223
2230499337,12cyclictest7950irq/56-dwc_otg_11:00:052
2230499335,5cyclictest8050irq/56-dwc_otg_10:00:172
2230499335,19cyclictest12902-21ssh12:10:192
2230499334,23cyclictest8050irq/56-dwc_otg_09:50:172
2230499331,6cyclictest28047-21ssh11:08:412
22304993315,4cyclictest16906-21/usr/sbin/munin12:15:292
2230499331,15cyclictest0-21swapper/210:06:302
2230399337,15cyclictest0-21swapper/112:25:231
2230399334,14cyclictest12870-21ssh10:50:061
2230399333,16cyclictest8050irq/56-dwc_otg_08:50:051
2230399332,3cyclictest0-21swapper/110:55:241
2230399332,20cyclictest0-21swapper/111:43:351
2230399332,20cyclictest0-21swapper/111:43:351
22303993312,7cyclictest10134-21unixbench_multi11:25:351
2230399331,25cyclictest0-21swapper/107:40:311
2230399331,17cyclictest7483-21ssh12:03:091
2230399331,13cyclictest0-21swapper/109:31:231
2230599322,24cyclictest8050irq/56-dwc_otg_08:30:203
2230499329,7cyclictest2084-21/usr/sbin/munin09:55:342
2230499328,10cyclictest7950irq/56-dwc_otg_12:00:242
2230499323,7cyclictest7950irq/56-dwc_otg_08:55:252
2230499323,5cyclictest29611-21wget07:30:192
22304993213,4cyclictest8050irq/56-dwc_otg_09:10:232
2230399328,14cyclictest20624-21idleruntime-cro12:20:041
2230399327,4cyclictest0-21swapper/109:55:321
2230399326,6cyclictest7850irq/56-dwc_otg08:20:331
2230399326,23cyclictest0-21swapper/107:30:191
22303993217,5cyclictest32612-21systemd-journal08:09:111
2230399321,5cyclictest0-21swapper/112:36:081
2230399321,5cyclictest0-21swapper/112:36:081
2230399321,14cyclictest9093-21unixbench_singl08:00:341
22303993211,4cyclictest0-21swapper/110:15:261
22303993210,12cyclictest0-21swapper/109:24:141
2230499317,13cyclictest20944-21apt-get12:20:182
2230499313,6cyclictest0-21swapper/207:10:182
2230499313,15cyclictest0-21swapper/207:20:052
2230499313,14cyclictest8050irq/56-dwc_otg_11:38:522
2230499311,20cyclictest2287-21latency_hist07:45:052
2230499311,17cyclictest532-21ls07:40:052
22304993110,6cyclictest31-21ksoftirqd/211:30:232
2230399319,9cyclictest1135-21cut09:10:211
2230399318,12cyclictest689-21snmpd08:15:261
2230399318,12cyclictest689-21snmpd08:15:261
2230399317,22cyclictest0-21swapper/108:25:201
2230399315,14cyclictest7950irq/56-dwc_otg_08:50:251
22303993118,5cyclictest661-21runrttasks08:40:211
22303993116,7cyclictest1200-21fschecks_time10:35:251
2230399311,4cyclictest7850irq/56-dwc_otg11:45:331
22303993114,4cyclictest7950irq/56-dwc_otg_09:40:231
2230399311,23cyclictest0-21swapper/108:55:561
2230399311,17cyclictest9701-21/usr/sbin/munin10:05:351
2230599307,3cyclictest8050irq/56-dwc_otg_07:59:103
2230599307,3cyclictest8050irq/56-dwc_otg_07:59:103
2230499308,8cyclictest301ktimersoftd/208:20:212
2230499302,8cyclictest7850irq/56-dwc_otg12:05:332
2230399309,9cyclictest30714-21aten_rbpower_cu09:50:181
2230399309,4cyclictest12994-21netstat09:25:301
2230399304,21cyclictest0-21swapper/107:15:181
2230399301,4cyclictest0-21swapper/109:10:061
2230399301,25cyclictest2696-21ls07:45:211
2230399301,19cyclictest0-21swapper/108:30:221
2230499292,16cyclictest0-21swapper/209:02:572
2230499291,17cyclictest10033-21cut08:05:272
206552290,3sleep20-21swapper/207:05:232
206552290,3sleep20-21swapper/207:05:232
2230599282,6cyclictest31340-21/usr/sbin/munin07:35:303
2230399286,8cyclictest0-21swapper/108:35:181
211422285,10sleep10-21swapper/107:05:311
211422285,10sleep10-21swapper/107:05:311
2230499271,5cyclictest26435-21grep08:50:262
220132273,15sleep30-21swapper/307:09:543
220132273,15sleep30-21swapper/307:09:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional