You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 09:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri May 22, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21412996945,14cyclictest11427-21ssh23:00:121
2141299614,42cyclictest24114-21ssh23:25:141
2141199574,41cyclictest11363-21ssh00:30:130
2141199551,42cyclictest9626-21ssh22:05:110
21427995316,18cyclictest20479-21ssh23:05:142
2141199525,4cyclictest131rcu_preempt23:25:140
2142799512,23cyclictest7889-21ssh00:10:122
21412995131,7cyclictest21-21ksoftirqd/123:10:121
21412995131,7cyclictest21-21ksoftirqd/123:10:111
21427994829,6cyclictest22278-21apt-get00:00:142
21427994728,15cyclictest25904-21apt-get00:20:132
21427994534,4cyclictest30270-21apt-get21:00:122
2141299453,7cyclictest131rcu_preempt22:55:141
2142799441,27cyclictest31382-21ssh22:35:142
2141299442,4cyclictest131rcu_preempt00:05:121
2141299442,4cyclictest131rcu_preempt00:05:111
21435994330,7cyclictest35966irq/58-eth022:00:123
21427994329,10cyclictest13349-21apt-get22:25:142
21427994230,8cyclictest4926-21apt-get20:00:122
21427994228,10cyclictest11353-21apt-get21:30:142
2141299423,8cyclictest131rcu_preempt19:35:111
2141199423,27cyclictest29458-21ssh23:10:120
2141199423,27cyclictest29458-21ssh23:10:110
21427994112,24cyclictest0-21swapper/221:20:142
2141299413,30cyclictest2470-21ssh00:25:151
2141299413,28cyclictest29500-21ssh21:40:111
2141199415,3cyclictest131rcu_preempt21:10:150
2141199412,29cyclictest18659-21ssh22:10:130
2143599403,24cyclictest35966irq/58-eth000:35:143
21412994022,7cyclictest21-21ksoftirqd/121:35:141
2141199402,27cyclictest16871-21ssh22:45:130
2141199402,27cyclictest16871-21ssh22:45:120
2143599393,22cyclictest35966irq/58-eth022:30:123
2142799391,29cyclictest20453-21sh21:35:142
21411993915,18cyclictest12-21ksoftirqd/023:45:130
21427993826,7cyclictest642-21apt-get23:30:122
21427993822,12cyclictest4309-21apt-get22:20:112
21427993819,15cyclictest16701-21ssh21:15:152
21427993815,12cyclictest0-21swapper/221:10:132
21427993810,17cyclictest9571-21ssh21:28:212
21412993831,3cyclictest21-21ksoftirqd/120:00:111
2141199381,27cyclictest22371-21ssh22:30:120
2142799379,19cyclictest7607-21ssh22:03:022
2142799379,19cyclictest1282-21rm23:46:562
2142799375,25cyclictest0-21swapper/200:30:142
21427993714,18cyclictest0-21swapper/223:35:142
2141199374,8cyclictest12-21ksoftirqd/023:35:120
2143599364,20cyclictest35966irq/58-eth021:15:133
2142799369,9cyclictest18037-21ssh22:09:452
2142799369,16cyclictest7241-21diskmemload22:18:252
2142799367,14cyclictest383-21irqcore19:55:192
2142799365,14cyclictest16988-21ssh00:15:142
21427993614,18cyclictest21410-21cyclictest23:10:122
21427993614,18cyclictest21410-21cyclictest23:10:112
21427993612,17cyclictest930-21snmpd20:20:102
21427993610,13cyclictest7241-21diskmemload23:19:472
2141299361,31cyclictest13369-21ssh22:25:141
2141199365,8cyclictest12-21ksoftirqd/021:20:130
2143599351,18cyclictest35966irq/58-eth022:45:133
2143599351,18cyclictest35966irq/58-eth022:45:123
2142799353,23cyclictest27-21ksoftirqd/221:40:132
21427993527,3cyclictest14602-21apt-get20:10:122
21412993518,13cyclictest21-21ksoftirqd/122:00:121
2141199354,22cyclictest24144-21ssh21:55:160
2141199354,22cyclictest24144-21ssh21:55:150
21435993423,5cyclictest35966irq/58-eth023:00:133
2143599341,24cyclictest35966irq/58-eth019:55:143
2142799349,10cyclictest7241-21diskmemload00:06:002
2142799349,10cyclictest7241-21diskmemload00:05:592
2142799346,20cyclictest0-21swapper/222:54:092
2142799345,11cyclictest91-21pr/ttyS020:40:022
2142799342,25cyclictest16875-21ssh22:45:132
2142799342,25cyclictest16875-21ssh22:45:122
21427993417,12cyclictest0-21swapper/223:50:122
21412993419,9cyclictest21410-21cyclictest23:40:141
2141199342,28cyclictest2344-21ssh21:25:130
2143599332,17cyclictest35966irq/58-eth021:55:163
2143599332,17cyclictest35966irq/58-eth021:55:153
2142799339,7cyclictest7241-21diskmemload22:41:402
2142799339,10cyclictest7241-21diskmemload23:28:272
2142799338,7cyclictest7241-21diskmemload23:04:442
2142799337,18cyclictest12987-21ssh21:47:562
2142799337,18cyclictest0-21swapper/221:50:192
2142799336,18cyclictest0-21swapper/223:59:142
2142799336,11cyclictest6173-21ps21:05:272
2142799335,14cyclictest23664-21cat20:15:302
21427993324,6cyclictest3701-21apt-get19:25:152
21427993320,9cyclictest20299-21ssh00:35:112
21427993315,13cyclictest0-21swapper/219:35:132
2142799331,23cyclictest15067-21ssh23:20:142
21427993310,8cyclictest7241-21diskmemload00:25:102
2141299333,18cyclictest6052-21diskmemload23:15:121
2141199333,21cyclictest648-21ssh23:30:130
2142799328,9cyclictest27021-21diskmemload22:14:432
2142799326,17cyclictest0-21swapper/223:40:482
2142799325,10cyclictest0-21swapper/222:33:452
2141199328,4cyclictest131rcu_preempt23:55:140
2141199321,18cyclictest27657-21ssh22:15:110
2142799317,15cyclictest0-21swapper/221:58:462
2142799317,15cyclictest0-21swapper/221:58:452
2142799314,13cyclictest26409-21/usr/sbin/munin19:45:262
2142799314,11cyclictest0-21swapper/219:10:172
21427993114,9cyclictest9803-21diskstats20:05:142
2143599305,17cyclictest35966irq/58-eth021:10:153
2143599304,16cyclictest35966irq/58-eth000:10:133
21435993013,10cyclictest0-21swapper/319:45:123
2142799305,13cyclictest9761-21ssh22:58:312
21427993024,3cyclictest29060-21apt-get20:25:132
21427993010,16cyclictest31346-21fschecks_count19:20:292
21427992912,8cyclictest29079-21sed20:55:272
21427992910,8cyclictest17604-21grep20:45:202
2141299293,21cyclictest7897-21ssh22:40:121
2141299291,21cyclictest25873-21ssh22:50:131
2141199294,16cyclictest12-21ksoftirqd/020:45:110
2143599282,16cyclictest35966irq/58-eth022:25:133
21427992811,9cyclictest21580-21ntp_states19:40:232
2141299282,19cyclictest22395-21ssh22:30:131
2141299281,18cyclictest25914-21ssh00:20:121
21412992810,5cyclictest131rcu_preempt21:30:131
2141199285,11cyclictest0-21swapper/020:00:120
21435992722,3cyclictest35966irq/58-eth023:10:133
21435992722,3cyclictest35966irq/58-eth023:10:123
21427992718,4cyclictest1420-21apt-get20:30:112
21427992711,12cyclictest12120-21grep19:30:262
21427992710,8cyclictest28210-21/usr/sbin/munin19:50:162
21427992710,8cyclictest25186-21cat20:50:292
2141199272,19cyclictest32697-21ssh00:22:510
2141199272,18cyclictest7907-21sh00:10:140
2143599262,14cyclictest35966irq/58-eth000:25:143
2141299265,16cyclictest18681-21rm22:10:131
2141299262,19cyclictest23661-21ssh00:17:501
2141199262,19cyclictest9427-21ssh00:28:070
21411992613,5cyclictest536-21usb-storage20:10:130
21411992611,8cyclictest11365-21ssh21:30:130
2141299259,11cyclictest21-21ksoftirqd/123:52:471
2141199252,18cyclictest23089-21ssh00:17:080
21411992512,4cyclictest12-21ksoftirqd/020:50:260
21411992510,7cyclictest12-21ksoftirqd/021:00:160
21435992410,9cyclictest35966irq/58-eth000:30:143
2142799247,13cyclictest15613-21latency_hist20:45:012
2141299245,9cyclictest0-21swapper/119:10:131
2141199242,16cyclictest12-21ksoftirqd/020:20:120
2143599231,18cyclictest35966irq/58-eth022:05:113
2142799231,14cyclictest0-21swapper/219:15:132
2141299231,17cyclictest628-21tcf-agent20:25:111
21412992311,3cyclictest131rcu_preempt21:50:181
21412992310,8cyclictest0-21swapper/100:00:141
2141199239,5cyclictest12-21ksoftirqd/019:35:150
2141199231,3cyclictest131rcu_preempt19:25:120
2141199231,15cyclictest4342-21sh22:20:140
21411992310,3cyclictest12-21ksoftirqd/022:40:250
2143599222,13cyclictest35966irq/58-eth000:15:123
21435992217,3cyclictest321rcuc/321:44:193
21435992215,5cyclictest0-21swapper/322:35:023
21435992214,5cyclictest0-21swapper/322:57:093
2141299222,15cyclictest20302-21ssh23:58:081
2141299221,4cyclictest20452-21apt-get23:05:141
2141199221,4cyclictest0-21swapper/021:15:190
21411992213,7cyclictest12-21ksoftirqd/021:45:190
21411992212,7cyclictest23822-21ssh21:55:010
2141199221,18cyclictest31461-21ls19:50:260
2141199221,18cyclictest21868-21ls19:10:170
2141199221,15cyclictest24075-21ssh21:35:260
2143599219,8cyclictest35966irq/58-eth000:05:133
2143599219,8cyclictest35966irq/58-eth000:05:123
2143599214,9cyclictest0-21swapper/321:30:113
2143599212,13cyclictest0-21swapper/323:25:143
21435992115,4cyclictest0-21swapper/323:18:123
21435992115,4cyclictest0-21swapper/322:42:543
21435992114,4cyclictest0-21swapper/321:45:273
2141299218,10cyclictest30606-21irqcore21:00:181
2141299212,14cyclictest659-21ssh23:30:141
21412992112,6cyclictest0-21swapper/100:35:191
2141299210,19cyclictest16493-21ls00:15:011
2141299210,17cyclictest16781-21ls21:15:161
2141299210,11cyclictest28785-21kthreadcore23:45:181
2141199219,4cyclictest12-21ksoftirqd/019:30:200
2141199219,3cyclictest131rcu_preempt23:15:300
2141199219,3cyclictest12-21ksoftirqd/019:45:170
2141199211,18cyclictest4169-21ls22:00:230
2141199211,17cyclictest6313-21ls22:55:290
2143599203,11cyclictest35966irq/58-eth023:45:093
21435992014,4cyclictest0-21swapper/323:35:123
21435992014,4cyclictest0-21swapper/323:30:143
21435992014,4cyclictest0-21swapper/321:50:153
21435992014,4cyclictest0-21swapper/321:20:303
21435992013,5cyclictest0-21swapper/323:41:163
21435992013,5cyclictest0-21swapper/323:06:073
21435992013,5cyclictest0-21swapper/321:25:153
21435992012,5cyclictest0-21swapper/322:15:263
21435992011,7cyclictest0-21swapper/321:35:203
2141299201,8cyclictest11733-21unixbench_multi21:10:301
21412992014,4cyclictest0-21swapper/120:55:271
21412992012,5cyclictest0-21swapper/120:45:251
21412992010,7cyclictest8568-21df_inode19:30:151
2141299200,11cyclictest2682-21kthreadcore20:30:191
2141199209,3cyclictest12-21ksoftirqd/023:50:250
2141199209,3cyclictest12-21ksoftirqd/020:05:100
2141199209,3cyclictest12-21ksoftirqd/000:00:140
2141199203,4cyclictest131rcu_preempt21:40:110
2141199202,14cyclictest2570-21ssh22:35:260
21411992014,4cyclictest0-21swapper/023:40:120
2141199200,17cyclictest17399-21ls22:25:290
2143599192,15cyclictest0-21swapper/300:20:123
2143599192,11cyclictest0-21swapper/321:00:153
2143599192,11cyclictest0-21swapper/320:20:133
2143599192,10cyclictest0-21swapper/321:05:133
2143599192,10cyclictest0-21swapper/320:00:153
21435991913,4cyclictest0-21swapper/323:55:143
21435991913,4cyclictest0-21swapper/322:20:183
21435991913,4cyclictest0-21swapper/322:10:153
21435991913,4cyclictest0-21swapper/300:03:513
21435991911,6cyclictest0-21swapper/322:50:033
2143599191,13cyclictest0-21swapper/320:10:163
2143599191,13cyclictest0-21swapper/319:15:113
21435991910,6cyclictest0-21swapper/323:50:133
2142799192,13cyclictest21558-21latency_hist19:10:022
2141299198,8cyclictest16473-21cron22:45:011
2141299198,8cyclictest16473-21cron22:45:011
2141299196,10cyclictest26254-21irqrtprio21:20:181
2141299192,5cyclictest53646usb-storage22:20:111
2141299192,13cyclictest27757-21ntp_192.168.11421:55:281
2141299192,13cyclictest27757-21ntp_192.168.11421:55:271
21412991912,4cyclictest0-21swapper/121:45:121
2141299191,13cyclictest22925-21apt-get19:45:131
21412991910,7cyclictest4671-21cron20:00:011
2141199199,2cyclictest12-21ksoftirqd/020:15:210
2141199199,2cyclictest12-21ksoftirqd/000:35:280
2141199198,3cyclictest131rcu_preempt20:30:020
2141199197,8cyclictest12-21ksoftirqd/019:40:110
2141199197,3cyclictest12-21ksoftirqd/023:10:010
2141199197,3cyclictest12-21ksoftirqd/000:05:120
2141199197,3cyclictest12-21ksoftirqd/000:05:110
21411991913,4cyclictest0-21swapper/020:40:180
21411991913,4cyclictest0-21swapper/019:55:170
21411991912,5cyclictest16527-21ssh23:00:470
21411991912,4cyclictest12-21ksoftirqd/019:20:360
21411991911,5cyclictest12-21ksoftirqd/022:50:120
21411991910,7cyclictest20929-21ssh23:21:430
2141199190,14cyclictest1742-21sed20:30:170
2143599182,7cyclictest0-21swapper/320:15:123
2143599182,10cyclictest0-21swapper/320:40:133
21435991812,4cyclictest0-21swapper/323:20:163
21435991812,4cyclictest0-21swapper/319:35:153
2143599181,13cyclictest0-21swapper/319:10:233
21435991810,5cyclictest0-21swapper/320:35:133
2141299188,7cyclictest19412-21apt-get20:15:121
2141299184,5cyclictest1457-21kthreadcore19:20:361
2141299182,12cyclictest0-21swapper/119:25:141
2141299182,11cyclictest844-21munin-node22:35:151
2141299181,7cyclictest18374-21latency_hist23:40:011
2141299181,15cyclictest18139-21ls19:40:121
2141299181,14cyclictest9709-21apt-get20:05:101
2141299181,14cyclictest15221-21mailstats00:30:281
2141199182,6cyclictest21426-21latency_hist19:10:010
21411991813,3cyclictest12-21ksoftirqd/019:15:170
21411991812,4cyclictest0-21swapper/020:35:140
2143599172,10cyclictest0-21swapper/320:50:133
21435991712,3cyclictest0-21swapper/319:40:203
21435991711,4cyclictest0-21swapper/320:55:153
21435991711,4cyclictest0-21swapper/320:30:173
21435991711,4cyclictest0-21swapper/319:30:173
21435991711,4cyclictest0-21swapper/319:25:163
21435991710,4cyclictest0-21swapper/319:50:203
2141299177,2cyclictest0-21swapper/122:15:121
2141299173,11cyclictest18426-21grep20:10:271
2141299171,13cyclictest12907-21kthreadcore20:40:201
2141299171,12cyclictest2451-21ssh21:25:161
2141299171,12cyclictest16880-21tr23:20:211
2141199178,7cyclictest28761-21memory20:55:210
21435991611,3cyclictest0-21swapper/320:25:243
21435991610,4cyclictest0-21swapper/319:20:393
2141299168,6cyclictest5666-21tr21:05:211
2141299162,10cyclictest10747-21tr20:40:011
2141299162,10cyclictest0-21swapper/119:50:111
2141299161,12cyclictest9906-21sh22:05:151
21412991610,4cyclictest28119-21unixbench_singl20:20:271
2141299160,13cyclictest28934-21kthreadcore19:15:201
2141199162,10cyclictest0-21swapper/021:05:130
2143599159,4cyclictest0-21swapper/320:05:173
2143599151,8cyclictest0-21swapper/320:45:123
21412991510,3cyclictest20636-21acpi20:50:111
2141299141,10cyclictest21410-21cyclictest19:10:011
2143599137,4cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional